2024-03-16T09:20:14.942 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T09:20:14.942 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339 --verbose --name pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/orig.config.yaml 2024-03-16T09:20:14.999 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T09:20:15.083 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} email: pdonnell@redhat.com first_in_suite: false job_id: '7602339' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLTS8r9NdLFvN0K4BbM6uhE+yI9Nh0KNrEGT9++u0nHPXqlYpk/PUO4lBpNqgbd0oEbrzFrhywZxP2HhLpnwdBM= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBIe/wTNnCVhVXqDA4I9ZuwtHYwjcOZZNsln7CQLWslzBgc6wxMD0ro2uDDtUQ4MIdroV0ko3yVkEbTt1jSivM= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn+HU31qDJ+LsuVLv+K9fHW3uyynuHEGkCeum5Z5T5qfWVPTL7G/mSYcHZ6fMPiBCF6H+MCPYnEhKcnx8mNWTY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_13:00:41 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T09:20:15.083 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa; will attempt to use it 2024-03-16T09:20:15.084 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks 2024-03-16T09:20:15.084 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T09:20:15.085 INFO:teuthology.task.internal:Checking packages... 2024-03-16T09:20:15.110 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-16T09:20:15.110 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T09:20:15.110 INFO:teuthology.packaging:ref: None 2024-03-16T09:20:15.111 INFO:teuthology.packaging:tag: None 2024-03-16T09:20:15.111 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-16T09:20:15.111 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:20:15.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-16T09:20:15.332 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-16T09:20:15.334 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T09:20:15.340 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T09:20:15.340 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T09:20:15.367 INFO:teuthology.task.internal:Saving configuration 2024-03-16T09:20:15.387 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T09:20:15.393 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T09:20:15.416 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:15:11.667351', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLTS8r9NdLFvN0K4BbM6uhE+yI9Nh0KNrEGT9++u0nHPXqlYpk/PUO4lBpNqgbd0oEbrzFrhywZxP2HhLpnwdBM='} 2024-03-16T09:20:15.440 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:15:11.665921', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBIe/wTNnCVhVXqDA4I9ZuwtHYwjcOZZNsln7CQLWslzBgc6wxMD0ro2uDDtUQ4MIdroV0ko3yVkEbTt1jSivM='} 2024-03-16T09:20:15.458 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:15:11.668586', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn+HU31qDJ+LsuVLv+K9fHW3uyynuHEGkCeum5Z5T5qfWVPTL7G/mSYcHZ6fMPiBCF6H+MCPYnEhKcnx8mNWTY='} 2024-03-16T09:20:15.459 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T09:20:15.466 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-16T09:20:15.466 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-16T09:20:15.466 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-16T09:20:15.466 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T09:20:15.559 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f46dd855160>, signals=[15]) 2024-03-16T09:20:15.559 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T09:20:15.565 INFO:teuthology.task.internal:Opening connections... 2024-03-16T09:20:15.565 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:20:15.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:20:15.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-03-16T09:20:15.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:20:15.747 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-03-16T09:20:15.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:20:15.837 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T09:20:15.843 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-03-16T09:20:15.861 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-03-16T09:20:15.861 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-03-16T09:20:15.918 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-03-16T09:20:15.918 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-03-16T09:20:15.918 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-03-16T09:20:15.918 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-03-16T09:20:15.918 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-03-16T09:20:15.918 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-03-16T09:20:15.918 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T09:20:15.919 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-03-16T09:20:15.919 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-03-16T09:20:15.919 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T09:20:15.919 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-03-16T09:20:15.919 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T09:20:15.921 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T09:20:15.921 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T09:20:15.922 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-03-16T09:20:15.944 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-03-16T09:20:15.962 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-03-16T09:20:15.962 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-03-16T09:20:16.019 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-03-16T09:20:16.019 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-03-16T09:20:16.019 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T09:20:16.020 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T09:20:16.020 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-03-16T09:20:16.036 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-03-16T09:20:16.053 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-03-16T09:20:16.053 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-03-16T09:20:16.109 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-03-16T09:20:16.109 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-03-16T09:20:16.109 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T09:20:16.110 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T09:20:16.110 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-03-16T09:20:16.127 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T09:20:16.138 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T09:20:16.144 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T09:20:16.144 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-03-16T09:20:16.147 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-03-16T09:20:16.149 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-03-16T09:20:16.165 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T09:20:16.173 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T09:20:16.173 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-03-16T09:20:16.206 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-03-16T09:20:16.209 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-03-16T09:20:16.241 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T09:20:16.334 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T09:20:16.377 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-16T09:20:16.378 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-16T09:20:16.378 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T09:20:16.378 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T09:20:16.379 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T09:20:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:20:16.397 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-03-16T09:20:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:20:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:20:16.399 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-16T09:20:16.399 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-16T09:20:16.455 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2024-03-16T09:20:16.456 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2024-03-16T09:20:16.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-03-16T09:20:16.457 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-361.el9.x86_64 2024-03-16T09:20:16.457 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.14.0-361.el9.x86_64 2024-03-16T09:20:16.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-03-16T09:20:16.458 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-361.el9.x86_64 2024-03-16T09:20:16.459 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.14.0-361.el9.x86_64 2024-03-16T09:20:16.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-03-16T09:20:18.498 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-16T09:20:18.710 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-16T09:20:18.716 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-16T09:20:19.882 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-16T09:20:20.102 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-16T09:20:20.117 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-16T09:20:21.024 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-16T09:20:22.010 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-03-16T09:20:22.347 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-16T09:20:23.000 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-16T09:20:24.098 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-16T09:20:27.901 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-16T09:20:29.419 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-03-16T09:20:30.460 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 19 MB 00:07 2024-03-16T09:20:30.681 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-16T09:20:34.419 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-16T09:20:35.653 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-16T09:20:35.886 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-03-16T09:20:36.140 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 732 kB/s | 6.1 MB 00:08 2024-03-16T09:20:36.654 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 32 MB/s | 21 MB 00:00 2024-03-16T09:20:37.498 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 15 kB 00:00 2024-03-16T09:20:37.818 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T09:20:37.819 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T09:20:37.909 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T09:20:37.920 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T09:20:37.921 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:20:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-03-16T09:20:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-16T09:20:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:Install 4 Packages 2024-03-16T09:20:37.921 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:20:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 100 M 2024-03-16T09:20:37.921 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 123 M 2024-03-16T09:20:37.922 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-03-16T09:20:38.480 INFO:teuthology.orchestra.run.smithi077.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-03-16T09:20:38.930 INFO:teuthology.orchestra.run.smithi077.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-03-16T09:20:39.764 INFO:teuthology.orchestra.run.smithi077.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 26 MB/s | 33 MB 00:01 2024-03-16T09:20:40.106 INFO:teuthology.orchestra.run.smithi077.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-03-16T09:20:40.106 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:20:40.106 INFO:teuthology.orchestra.run.smithi077.stdout:Total 46 MB/s | 100 MB 00:02 2024-03-16T09:20:40.988 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-03-16T09:20:41.369 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-03-16T09:20:41.369 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-03-16T09:20:41.604 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-16T09:20:42.697 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-03-16T09:20:42.697 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-03-16T09:20:43.794 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T09:20:43.794 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T09:20:43.904 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-16T09:20:43.913 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-16T09:20:43.913 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-16T09:20:43.913 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-16T09:20:43.913 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-16T09:20:43.913 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-03-16T09:20:43.914 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:20:43.915 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 100 M 2024-03-16T09:20:43.915 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-03-16T09:20:43.915 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-16T09:20:44.447 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-03-16T09:20:45.022 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-03-16T09:20:45.415 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-03-16T09:20:45.564 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 30 MB/s | 33 MB 00:01 2024-03-16T09:20:46.056 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:20:46.190 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:20:47.140 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 13 MB/s | 39 MB 00:03 2024-03-16T09:20:47.143 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:20:47.143 INFO:teuthology.orchestra.run.smithi129.stdout:Total 31 MB/s | 100 MB 00:03 2024-03-16T09:20:48.034 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-16T09:20:48.390 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-16T09:20:48.391 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-16T09:20:49.000 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:20:49.177 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:20:49.734 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-16T09:20:49.734 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-16T09:20:52.667 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-16T09:20:53.319 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:20:53.465 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:20:53.914 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:20:54.037 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:20:56.236 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:20:56.432 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:20:57.884 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-16T09:20:58.777 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:01.172 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:21:01.384 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:02.855 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-16T09:21:05.095 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T09:21:05.096 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T09:21:05.186 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T09:21:05.195 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T09:21:05.196 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:21:05.196 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-16T09:21:05.196 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-16T09:21:05.196 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-03-16T09:21:05.196 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:21:05.196 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 100 M 2024-03-16T09:21:05.197 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 123 M 2024-03-16T09:21:05.197 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-16T09:21:06.111 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:13.831 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 735 kB/s | 6.0 MB 00:08 2024-03-16T09:21:28.122 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:28.122 INFO:teuthology.orchestra.run.smithi077.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T09:21:28.122 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:21:28.206 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:28.594 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:28.594 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:21:28.594 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:21:28.594 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:21:29.015 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-03-16T09:21:29.489 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:21:29.634 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:21:29.634 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:21:29.634 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T09:21:29.635 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T09:21:29.635 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T09:21:29.635 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-16T09:21:29.635 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T09:21:29.635 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:21:29.755 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:21:29.755 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:21:29.756 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2024-03-16T09:21:29.789 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.06 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 68.el9 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Group : Unspecified 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 8264254 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : builder@centos.org 2024-03-16T09:21:29.790 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:21:29.791 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T09:21:29.792 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T09:21:29.792 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 5.14.0-427.el9.x86_64 2024-03-16T09:21:29.792 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T09:21:30.429 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-03-16T09:21:31.807 INFO:teuthology.orchestra.run.smithi077.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T09:21:31.837 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-03-16T09:21:31.838 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-03-16T09:21:31.853 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.BMwaKkyYb4 2024-03-16T09:21:31.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BMwaKkyYb4 2024-03-16T09:21:31.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.BMwaKkyYb4 2024-03-16T09:21:32.056 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.BMwaKkyYb4 is 6KB 2024-03-16T09:21:32.104 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.BMwaKkyYb4 2024-03-16T09:21:32.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T09:21:32.192 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T09:21:32.192 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T09:21:32.192 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T09:21:32.193 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T09:21:32.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T09:21:32.379 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-03-16T09:21:34.870 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 726 kB/s | 21 MB 00:29 2024-03-16T09:21:35.299 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:35.299 INFO:teuthology.orchestra.run.smithi129.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T09:21:35.299 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:21:35.383 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:35.808 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:35.808 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:21:35.808 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:21:35.808 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:21:36.224 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-16T09:21:36.727 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:21:36.867 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:21:36.867 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:21:36.868 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T09:21:36.868 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T09:21:36.868 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T09:21:36.868 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-16T09:21:36.868 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T09:21:36.868 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:21:36.989 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:21:36.989 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:21:36.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-03-16T09:21:37.023 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-03-16T09:21:37.023 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-03-16T09:21:37.023 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-03-16T09:21:37.023 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-03-16T09:21:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:21:37.025 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T09:21:37.025 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T09:21:37.025 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T09:21:37.025 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-03-16T09:21:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:21:37.025 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T09:21:37.026 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T09:21:37.026 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-427.el9.x86_64 2024-03-16T09:21:37.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T09:21:37.662 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-03-16T09:21:39.182 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T09:21:39.203 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-03-16T09:21:39.206 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-16T09:21:39.261 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.LmtPbuF2CM 2024-03-16T09:21:39.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LmtPbuF2CM 2024-03-16T09:21:39.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.LmtPbuF2CM 2024-03-16T09:21:39.472 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.LmtPbuF2CM is 6KB 2024-03-16T09:21:39.520 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.LmtPbuF2CM 2024-03-16T09:21:39.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T09:21:39.600 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T09:21:39.600 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T09:21:39.600 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T09:21:39.600 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T09:21:39.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T09:21:39.755 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-03-16T09:22:02.382 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-16T09:22:02.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:22:02.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:22:09.758 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-16T09:22:09.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-16T09:22:09.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:22:15.756 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 573 kB/s | 39 MB 01:10 2024-03-16T09:22:16.473 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 546 kB/s | 33 MB 01:02 2024-03-16T09:22:16.473 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:22:16.474 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.4 MB/s | 100 MB 01:11 2024-03-16T09:22:17.349 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-16T09:22:17.749 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-16T09:22:17.749 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-16T09:22:19.102 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-16T09:22:19.103 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-16T09:22:20.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-03-16T09:22:22.010 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-16T09:22:22.665 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:22:22.801 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:22:25.580 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:22:25.770 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:22:28.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-16T09:22:29.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:22:29.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:22:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:22:30.713 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:22:32.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-03-16T09:22:35.500 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:22:37.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-16T09:22:37.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:22:40.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-16T09:22:44.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:22:44.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:22:52.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-16T09:22:52.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:23:04.562 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:23:04.563 INFO:teuthology.orchestra.run.smithi119.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T09:23:04.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:23:04.647 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:23:05.107 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:23:05.107 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:23:05.107 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:23:05.107 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:23:05.530 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-16T09:23:06.042 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:23:06.238 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:23:06.239 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:23:06.239 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T09:23:06.239 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T09:23:06.239 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T09:23:06.239 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-16T09:23:06.239 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T09:23:06.239 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:23:06.360 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:23:06.360 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:23:06.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 68.el9 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8264254 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-03-16T09:23:06.398 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-03-16T09:23:06.399 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:23:06.400 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T09:23:06.400 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T09:23:06.400 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 5.14.0-427.el9.x86_64 2024-03-16T09:23:06.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T09:23:07.027 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-03-16T09:23:08.441 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T09:23:08.487 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-03-16T09:23:08.490 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-03-16T09:23:08.505 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.AcrxjwAIfm 2024-03-16T09:23:08.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AcrxjwAIfm 2024-03-16T09:23:08.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.AcrxjwAIfm 2024-03-16T09:23:08.706 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.AcrxjwAIfm is 6KB 2024-03-16T09:23:08.753 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.AcrxjwAIfm 2024-03-16T09:23:08.767 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T09:23:08.832 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T09:23:08.832 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T09:23:08.832 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T09:23:08.832 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T09:23:08.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T09:23:08.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-03-16T09:23:10.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-16T09:23:16.788 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-03-16T09:23:17.389 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:23:17.389 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-16T09:23:17.389 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-03-16T09:23:17.403 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-427.el9.x86_64 2024-03-16T09:23:17.403 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T09:23:17.403 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T09:23:17.403 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T09:23:18.403 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-16T09:23:18.404 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T09:23:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-03-16T09:23:18.546 DEBUG:teuthology.parallel:result is None 2024-03-16T09:23:25.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-16T09:23:25.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:23:26.043 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-16T09:23:26.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-16T09:23:26.489 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-16T09:23:26.489 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-16T09:23:26.503 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-427.el9.x86_64 2024-03-16T09:23:26.503 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T09:23:26.503 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T09:23:26.503 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T09:23:27.504 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-16T09:23:27.504 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T09:23:27.590 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-03-16T09:23:27.625 DEBUG:teuthology.parallel:result is None 2024-03-16T09:23:38.995 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-16T09:23:38.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-16T09:23:38.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:23:57.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-16T09:24:06.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-16T09:24:06.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:24:09.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-16T09:24:21.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-16T09:24:21.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:24:53.260 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-16T09:24:53.829 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-16T09:24:53.829 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-16T09:24:53.829 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-16T09:24:53.843 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-427.el9.x86_64 2024-03-16T09:24:53.843 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T09:24:53.843 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T09:24:53.843 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T09:24:54.844 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-16T09:24:54.844 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T09:24:54.924 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-03-16T09:24:54.961 DEBUG:teuthology.parallel:result is None 2024-03-16T09:24:54.961 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T09:24:54.971 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T09:24:54.971 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T09:24:54.974 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T09:24:54.976 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T09:24:54.995 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T09:24:55.002 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T09:24:55.011 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T09:24:55.011 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T09:24:55.032 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T09:24:55.034 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T09:24:55.097 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T09:24:55.104 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T09:24:55.104 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T09:24:55.106 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T09:24:55.109 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T09:24:55.150 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:24:55.159 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:24:55.164 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:24:55.173 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:24:55.304 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:24:55.318 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:24:55.319 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T09:24:55.325 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T09:24:55.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T09:24:55.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T09:24:55.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T09:24:55.361 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T09:24:55.369 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T09:24:55.369 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T09:24:55.399 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T09:24:55.402 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T09:24:55.418 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:24:55.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:24:55.549 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:24:55.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:24:55.676 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:24:55.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T09:24:55.744 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:24:55.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:24:55.835 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:24:55.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:24:55.952 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:24:55.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T09:24:56.018 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:24:56.062 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:24:56.132 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:24:56.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:24:56.220 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:24:56.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T09:24:56.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-03-16T09:24:56.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-03-16T09:24:56.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-03-16T09:24:56.340 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:24:56.340 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:24:56.373 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:24:56.835 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T09:24:56.844 INFO:teuthology.task.internal:Starting timer... 2024-03-16T09:24:56.844 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T09:24:56.861 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T09:24:56.875 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-16T09:24:56.875 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-03-16T09:24:56.931 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-03-16T09:24:56.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-03-16T09:24:57.000 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-03-16T09:24:57.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-03-16T09:24:57.061 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-03-16T09:24:57.063 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T09:24:57.063 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-03-16T09:24:57.082 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-03-16T09:24:57.082 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-03-16T09:24:57.105 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-03-16T09:24:57.105 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-03-16T09:24:57.131 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-03-16T09:24:57.131 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-03-16T09:24:57.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:24:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:24:57.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:24:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:24:57.187 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:24:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:24:57.216 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T09:24:57.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-03-16T09:24:57.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-03-16T09:24:57.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-03-16T09:24:57.333 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T09:24:57.343 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T09:24:57.359 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T09:24:57.360 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi119.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T09:29:06.363 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-03-16T09:29:06.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:29:06.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:29:06.459 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-03-16T09:29:06.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:29:06.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-16T09:29:06.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:29:06.635 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-16T09:29:06.713 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-16T09:29:06.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-16T09:29:06.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:29:06.806 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-16T09:29:06.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-16T09:29:06.885 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T09:29:06.895 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T09:29:06.895 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T09:29:06.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T09:29:06.898 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T09:29:06.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T09:29:06.900 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T09:29:06.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T09:29:06.928 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T09:29:06.932 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T09:29:06.944 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T09:29:06.950 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T09:29:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T09:29:06.973 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-03-16T09:29:06.974 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T09:29:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-03-16T09:29:06.987 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-03-16T09:29:06.995 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-03-16T09:29:07.003 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T09:29:07.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-03-16T09:29:07.012 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T09:29:07.018 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-03-16T09:29:07.019 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T09:29:07.028 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T09:29:07.035 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T09:29:07.051 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T09:29:07.078 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-16T09:29:07.082 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T09:29:07.082 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-16T09:29:07.082 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.082 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.082 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.083 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.088 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-16T09:29:07.092 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T09:29:07.093 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-16T09:29:07.093 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.093 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.093 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.093 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-16T09:29:07.107 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T09:29:07.107 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-16T09:29:07.107 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.107 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.107 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.108 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:29:07.108 INFO:teuthology.run_tasks:Running task install... 2024-03-16T09:29:07.116 DEBUG:teuthology.task.install:project ceph 2024-03-16T09:29:07.117 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-16T09:29:07.117 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-16T09:29:07.117 INFO:teuthology.task.install:Using flavor: default 2024-03-16T09:29:07.122 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-16T09:29:07.122 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T09:29:07.122 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T09:29:07.123 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:29:07.124 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T09:29:07.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:29:07.125 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T09:29:07.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:29:07.275 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T09:29:07.275 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T09:29:07.278 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T09:29:07.278 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T09:29:07.350 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T09:29:07.351 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T09:29:07.391 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T09:29:07.391 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:29:07.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T09:29:07.400 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T09:29:07.400 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:29:07.400 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T09:29:07.428 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T09:29:07.428 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T09:29:07.429 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T09:29:07.430 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T09:29:07.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T09:29:07.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T09:29:07.525 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T09:29:07.526 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:29:07.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T09:29:07.555 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T09:29:07.555 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T09:29:07.576 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T09:29:07.578 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T09:29:07.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T09:29:07.642 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-03-16T09:29:07.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-03-16T09:29:07.644 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-03-16T09:29:07.646 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-03-16T09:29:07.705 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T09:29:07.772 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-03-16T09:29:07.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-03-16T09:29:07.882 INFO:teuthology.orchestra.run.smithi129.stdout:74 files removed 2024-03-16T09:29:07.900 INFO:teuthology.orchestra.run.smithi077.stdout:74 files removed 2024-03-16T09:29:07.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T09:29:07.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T09:29:08.035 INFO:teuthology.orchestra.run.smithi119.stdout:74 files removed 2024-03-16T09:29:08.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T09:29:08.647 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 198 kB/s | 83 kB 00:00 2024-03-16T09:29:08.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 200 kB/s | 83 kB 00:00 2024-03-16T09:29:08.812 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 191 kB/s | 83 kB 00:00 2024-03-16T09:29:09.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-16T09:29:09.089 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-16T09:29:09.221 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-16T09:29:09.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-16T09:29:09.540 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-16T09:29:09.630 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-16T09:29:10.133 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-16T09:29:10.157 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-16T09:29:10.218 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-16T09:29:11.552 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-16T09:29:11.583 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-16T09:29:11.852 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-16T09:29:12.675 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-16T09:29:12.699 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-16T09:29:14.426 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-16T09:29:14.785 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-16T09:29:16.960 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-03-16T09:29:18.395 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-16T09:29:18.685 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-16T09:29:18.760 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-16T09:29:19.778 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 15 kB 00:00 2024-03-16T09:29:20.923 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-16T09:29:21.185 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-16T09:29:22.920 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-16T09:29:23.180 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-16T09:29:24.437 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-03-16T09:29:25.879 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-16T09:29:26.541 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-03-16T09:29:28.161 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-16T09:29:28.410 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T09:29:28.411 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T09:29:28.423 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T09:29:28.424 INFO:teuthology.orchestra.run.smithi129.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T09:29:28.425 INFO:teuthology.orchestra.run.smithi129.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T09:29:28.426 INFO:teuthology.orchestra.run.smithi129.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T09:29:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T09:29:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T09:29:28.431 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T09:29:28.512 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-16T09:29:28.537 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-16T09:29:28.537 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T09:29:28.538 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T09:29:28.539 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T09:29:28.540 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T09:29:28.541 INFO:teuthology.orchestra.run.smithi129.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-16T09:29:28.542 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T09:29:28.543 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T09:29:28.544 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T09:29:28.545 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T09:29:28.546 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T09:29:28.547 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T09:29:28.548 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T09:29:28.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T09:29:28.550 INFO:teuthology.orchestra.run.smithi129.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout:Install 150 Packages 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 24 Packages 2024-03-16T09:29:28.551 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:29:28.552 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 224 M 2024-03-16T09:29:28.552 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-16T09:29:29.383 INFO:teuthology.orchestra.run.smithi129.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-16T09:29:29.883 INFO:teuthology.orchestra.run.smithi129.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.7 MB/s | 890 kB 00:00 2024-03-16T09:29:29.992 INFO:teuthology.orchestra.run.smithi129.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.3 MB/s | 145 kB 00:00 2024-03-16T09:29:30.058 INFO:teuthology.orchestra.run.smithi129.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 6.0 MB/s | 5.4 MB 00:00 2024-03-16T09:29:30.217 INFO:teuthology.orchestra.run.smithi129.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-16T09:29:30.609 INFO:teuthology.orchestra.run.smithi129.stdout:(6/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 12 MB/s | 4.7 MB 00:00 2024-03-16T09:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout:(7/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 3.6 MB/s | 2.4 MB 00:00 2024-03-16T09:29:30.667 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T09:29:30.667 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T09:29:30.682 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T09:29:30.683 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T09:29:30.684 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T09:29:30.685 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T09:29:30.687 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T09:29:30.690 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T09:29:30.691 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T09:29:30.772 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-16T09:29:30.791 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T09:29:30.792 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T09:29:30.793 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T09:29:30.794 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T09:29:30.795 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T09:29:30.796 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T09:29:30.797 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T09:29:30.798 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T09:29:30.799 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T09:29:30.800 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T09:29:30.801 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T09:29:30.802 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T09:29:30.803 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout:Install 150 Packages 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 24 Packages 2024-03-16T09:29:30.804 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:29:30.805 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 224 M 2024-03-16T09:29:30.805 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-16T09:29:31.497 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-16T09:29:31.754 INFO:teuthology.orchestra.run.smithi119.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-16T09:29:32.026 INFO:teuthology.orchestra.run.smithi129.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 7.2 MB/s | 21 MB 00:02 2024-03-16T09:29:32.201 INFO:teuthology.orchestra.run.smithi129.stdout:(9/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 11 MB/s | 17 MB 00:01 2024-03-16T09:29:32.218 INFO:teuthology.orchestra.run.smithi129.stdout:(10/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el 131 kB/s | 25 kB 00:00 2024-03-16T09:29:32.327 INFO:teuthology.orchestra.run.smithi129.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.0 MB/s | 221 kB 00:00 2024-03-16T09:29:32.387 INFO:teuthology.orchestra.run.smithi119.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-16T09:29:32.427 INFO:teuthology.orchestra.run.smithi129.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 271 kB/s | 27 kB 00:00 2024-03-16T09:29:32.471 INFO:teuthology.orchestra.run.smithi119.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.7 MB/s | 145 kB 00:00 2024-03-16T09:29:32.502 INFO:teuthology.orchestra.run.smithi129.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 424 kB/s | 32 kB 00:00 2024-03-16T09:29:32.563 INFO:teuthology.orchestra.run.smithi119.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 4.4 MB/s | 5.4 MB 00:01 2024-03-16T09:29:32.619 INFO:teuthology.orchestra.run.smithi129.stdout:(14/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 5.0 MB/s | 9.8 MB 00:01 2024-03-16T09:29:32.678 INFO:teuthology.orchestra.run.smithi129.stdout:(15/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.2 MB/s | 747 kB 00:00 2024-03-16T09:29:32.711 INFO:teuthology.orchestra.run.smithi129.stdout:(16/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.7 MB/s | 163 kB 00:00 2024-03-16T09:29:32.762 INFO:teuthology.orchestra.run.smithi129.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 127 kB 00:00 2024-03-16T09:29:32.878 INFO:teuthology.orchestra.run.smithi129.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.9 MB/s | 501 kB 00:00 2024-03-16T09:29:32.979 INFO:teuthology.orchestra.run.smithi129.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 451 kB/s | 45 kB 00:00 2024-03-16T09:29:33.104 INFO:teuthology.orchestra.run.smithi129.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.0 MB/s | 131 kB 00:00 2024-03-16T09:29:33.213 INFO:teuthology.orchestra.run.smithi129.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 162 kB 00:00 2024-03-16T09:29:33.380 INFO:teuthology.orchestra.run.smithi119.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-03-16T09:29:33.421 INFO:teuthology.orchestra.run.smithi129.stdout:(22/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 7.8 MB/s | 5.1 MB 00:00 2024-03-16T09:29:33.522 INFO:teuthology.orchestra.run.smithi119.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 1.6 MB/s | 1.5 MB 00:00 2024-03-16T09:29:33.530 INFO:teuthology.orchestra.run.smithi129.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.7 MB/s | 298 kB 00:00 2024-03-16T09:29:33.614 INFO:teuthology.orchestra.run.smithi129.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 99 kB 00:00 2024-03-16T09:29:33.714 INFO:teuthology.orchestra.run.smithi129.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 852 kB/s | 85 kB 00:00 2024-03-16T09:29:33.739 INFO:teuthology.orchestra.run.smithi129.stdout:(26/174): python3-rados-19.0.0-2162.g3d78f5ba.e 608 kB/s | 320 kB 00:00 2024-03-16T09:29:34.050 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T09:29:34.051 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T09:29:34.062 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T09:29:34.063 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T09:29:34.064 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T09:29:34.065 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T09:29:34.067 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T09:29:34.070 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T09:29:34.070 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T09:29:34.150 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-03-16T09:29:34.164 INFO:teuthology.orchestra.run.smithi129.stdout:(27/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 6.9 MB/s | 3.1 MB 00:00 2024-03-16T09:29:34.165 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-16T09:29:34.165 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-03-16T09:29:34.165 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-16T09:29:34.165 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-03-16T09:29:34.165 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T09:29:34.166 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T09:29:34.167 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T09:29:34.168 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T09:29:34.169 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T09:29:34.170 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T09:29:34.171 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T09:29:34.172 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T09:29:34.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T09:29:34.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T09:29:34.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T09:29:34.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T09:29:34.177 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout:Install 150 Packages 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 24 Packages 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:29:34.178 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 224 M 2024-03-16T09:29:34.179 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-03-16T09:29:34.256 INFO:teuthology.orchestra.run.smithi129.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 372 kB/s | 34 kB 00:00 2024-03-16T09:29:34.315 INFO:teuthology.orchestra.run.smithi129.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.3 MB/s | 139 kB 00:00 2024-03-16T09:29:34.415 INFO:teuthology.orchestra.run.smithi129.stdout:(30/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 252 kB/s | 170 kB 00:00 2024-03-16T09:29:34.447 INFO:teuthology.orchestra.run.smithi119.stdout:(7/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 6.6 MB/s | 21 MB 00:03 2024-03-16T09:29:34.547 INFO:teuthology.orchestra.run.smithi119.stdout:(8/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-03-16T09:29:34.566 INFO:teuthology.orchestra.run.smithi129.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 7.2 MB/s | 1.8 MB 00:00 2024-03-16T09:29:34.614 INFO:teuthology.orchestra.run.smithi119.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 378 kB/s | 25 kB 00:00 2024-03-16T09:29:34.649 INFO:teuthology.orchestra.run.smithi129.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.9 MB/s | 248 kB 00:00 2024-03-16T09:29:34.733 INFO:teuthology.orchestra.run.smithi129.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 593 kB/s | 49 kB 00:00 2024-03-16T09:29:34.825 INFO:teuthology.orchestra.run.smithi129.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 180 kB/s | 16 kB 00:00 2024-03-16T09:29:34.898 INFO:teuthology.orchestra.run.smithi077.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-03-16T09:29:34.933 INFO:teuthology.orchestra.run.smithi129.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.4 MB/s | 264 kB 00:00 2024-03-16T09:29:35.134 INFO:teuthology.orchestra.run.smithi129.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 3.7 MB/s | 748 kB 00:00 2024-03-16T09:29:35.217 INFO:teuthology.orchestra.run.smithi129.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 219 kB/s | 18 kB 00:00 2024-03-16T09:29:35.368 INFO:teuthology.orchestra.run.smithi129.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-16T09:29:35.418 INFO:teuthology.orchestra.run.smithi129.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-16T09:29:35.451 INFO:teuthology.orchestra.run.smithi129.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-16T09:29:35.493 INFO:teuthology.orchestra.run.smithi129.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-16T09:29:35.527 INFO:teuthology.orchestra.run.smithi129.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 955 kB/s | 32 kB 00:00 2024-03-16T09:29:35.619 INFO:teuthology.orchestra.run.smithi129.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-16T09:29:35.644 INFO:teuthology.orchestra.run.smithi129.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 690 kB/s | 17 kB 00:00 2024-03-16T09:29:35.648 INFO:teuthology.orchestra.run.smithi077.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 890 kB 00:00 2024-03-16T09:29:35.669 INFO:teuthology.orchestra.run.smithi129.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 400 kB/s | 10 kB 00:00 2024-03-16T09:29:35.703 INFO:teuthology.orchestra.run.smithi129.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-16T09:29:35.728 INFO:teuthology.orchestra.run.smithi129.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-16T09:29:35.795 INFO:teuthology.orchestra.run.smithi129.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-16T09:29:35.798 INFO:teuthology.orchestra.run.smithi077.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 965 kB/s | 145 kB 00:00 2024-03-16T09:29:35.837 INFO:teuthology.orchestra.run.smithi129.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-16T09:29:35.879 INFO:teuthology.orchestra.run.smithi129.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-16T09:29:35.921 INFO:teuthology.orchestra.run.smithi129.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-16T09:29:35.963 INFO:teuthology.orchestra.run.smithi129.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-16T09:29:35.997 INFO:teuthology.orchestra.run.smithi129.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2024-03-16T09:29:36.030 INFO:teuthology.orchestra.run.smithi129.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-16T09:29:36.164 INFO:teuthology.orchestra.run.smithi129.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-03-16T09:29:36.223 INFO:teuthology.orchestra.run.smithi129.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 26 MB/s | 1.5 MB 00:00 2024-03-16T09:29:36.265 INFO:teuthology.orchestra.run.smithi129.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 19 MB/s | 811 kB 00:00 2024-03-16T09:29:36.298 INFO:teuthology.orchestra.run.smithi129.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-16T09:29:36.423 INFO:teuthology.orchestra.run.smithi119.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 5.0 MB/s | 9.8 MB 00:01 2024-03-16T09:29:36.440 INFO:teuthology.orchestra.run.smithi129.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-16T09:29:36.499 INFO:teuthology.orchestra.run.smithi129.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-16T09:29:36.531 INFO:teuthology.orchestra.run.smithi119.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.0 MB/s | 221 kB 00:00 2024-03-16T09:29:36.532 INFO:teuthology.orchestra.run.smithi129.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-16T09:29:36.623 INFO:teuthology.orchestra.run.smithi119.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 294 kB/s | 27 kB 00:00 2024-03-16T09:29:36.666 INFO:teuthology.orchestra.run.smithi129.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 43 MB/s | 5.8 MB 00:00 2024-03-16T09:29:36.700 INFO:teuthology.orchestra.run.smithi129.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 737 kB/s | 24 kB 00:00 2024-03-16T09:29:36.707 INFO:teuthology.orchestra.run.smithi077.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-03-16T09:29:36.723 INFO:teuthology.orchestra.run.smithi119.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 317 kB/s | 32 kB 00:00 2024-03-16T09:29:36.733 INFO:teuthology.orchestra.run.smithi129.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 541 kB/s | 18 kB 00:00 2024-03-16T09:29:36.767 INFO:teuthology.orchestra.run.smithi129.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-16T09:29:36.801 INFO:teuthology.orchestra.run.smithi129.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 943 kB/s | 32 kB 00:00 2024-03-16T09:29:36.957 INFO:teuthology.orchestra.run.smithi119.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 3.1 MB/s | 747 kB 00:00 2024-03-16T09:29:36.959 INFO:teuthology.orchestra.run.smithi129.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-16T09:29:36.993 INFO:teuthology.orchestra.run.smithi129.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 6.4 MB/s | 217 kB 00:00 2024-03-16T09:29:37.026 INFO:teuthology.orchestra.run.smithi129.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-16T09:29:37.032 INFO:teuthology.orchestra.run.smithi119.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-16T09:29:37.060 INFO:teuthology.orchestra.run.smithi129.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-16T09:29:37.094 INFO:teuthology.orchestra.run.smithi129.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-16T09:29:37.116 INFO:teuthology.orchestra.run.smithi119.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 127 kB 00:00 2024-03-16T09:29:37.127 INFO:teuthology.orchestra.run.smithi129.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 7.0 MB/s | 238 kB 00:00 2024-03-16T09:29:37.161 INFO:teuthology.orchestra.run.smithi129.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-16T09:29:37.195 INFO:teuthology.orchestra.run.smithi129.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 6.2 MB/s | 211 kB 00:00 2024-03-16T09:29:37.228 INFO:teuthology.orchestra.run.smithi129.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-16T09:29:37.232 INFO:teuthology.orchestra.run.smithi077.stdout:(5/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-03-16T09:29:37.262 INFO:teuthology.orchestra.run.smithi129.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-16T09:29:37.266 INFO:teuthology.orchestra.run.smithi077.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 2.7 MB/s | 1.5 MB 00:00 2024-03-16T09:29:37.266 INFO:teuthology.orchestra.run.smithi119.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 3.3 MB/s | 501 kB 00:00 2024-03-16T09:29:37.579 INFO:teuthology.orchestra.run.smithi129.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-03-16T09:29:37.663 INFO:teuthology.orchestra.run.smithi129.stdout:(78/174): ceph-mgr-diskprediction-local-19.0.0- 2.3 MB/s | 7.4 MB 00:03 2024-03-16T09:29:37.679 INFO:teuthology.orchestra.run.smithi129.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 386 kB/s | 39 kB 00:00 2024-03-16T09:29:37.696 INFO:teuthology.orchestra.run.smithi129.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-16T09:29:37.722 INFO:teuthology.orchestra.run.smithi129.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-16T09:29:37.738 INFO:teuthology.orchestra.run.smithi129.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-16T09:29:37.756 INFO:teuthology.orchestra.run.smithi129.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-16T09:29:37.773 INFO:teuthology.orchestra.run.smithi129.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-16T09:29:37.798 INFO:teuthology.orchestra.run.smithi129.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-16T09:29:37.823 INFO:teuthology.orchestra.run.smithi129.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-16T09:29:37.857 INFO:teuthology.orchestra.run.smithi129.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-16T09:29:37.882 INFO:teuthology.orchestra.run.smithi129.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-16T09:29:37.907 INFO:teuthology.orchestra.run.smithi129.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-16T09:29:37.916 INFO:teuthology.orchestra.run.smithi119.stdout:(18/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 3.8 MB/s | 17 MB 00:04 2024-03-16T09:29:37.933 INFO:teuthology.orchestra.run.smithi129.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-16T09:29:38.000 INFO:teuthology.orchestra.run.smithi119.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 543 kB/s | 45 kB 00:00 2024-03-16T09:29:38.066 INFO:teuthology.orchestra.run.smithi129.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-16T09:29:38.083 INFO:teuthology.orchestra.run.smithi119.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-16T09:29:38.125 INFO:teuthology.orchestra.run.smithi129.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-03-16T09:29:38.142 INFO:teuthology.orchestra.run.smithi129.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-03-16T09:29:38.167 INFO:teuthology.orchestra.run.smithi129.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-03-16T09:29:38.167 INFO:teuthology.orchestra.run.smithi119.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-16T09:29:38.326 INFO:teuthology.orchestra.run.smithi119.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.0 MB/s | 320 kB 00:00 2024-03-16T09:29:38.476 INFO:teuthology.orchestra.run.smithi119.stdout:(23/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 4.2 MB/s | 5.1 MB 00:01 2024-03-16T09:29:38.501 INFO:teuthology.orchestra.run.smithi119.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 1.7 MB/s | 298 kB 00:00 2024-03-16T09:29:38.543 INFO:teuthology.orchestra.run.smithi119.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 99 kB 00:00 2024-03-16T09:29:38.549 INFO:teuthology.orchestra.run.smithi077.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 3.6 MB/s | 4.7 MB 00:01 2024-03-16T09:29:38.569 INFO:teuthology.orchestra.run.smithi119.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-16T09:29:38.660 INFO:teuthology.orchestra.run.smithi119.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-16T09:29:38.726 INFO:teuthology.orchestra.run.smithi129.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 37 kB/s | 21 kB 00:00 2024-03-16T09:29:38.736 INFO:teuthology.orchestra.run.smithi119.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 451 kB/s | 34 kB 00:00 2024-03-16T09:29:38.751 INFO:teuthology.orchestra.run.smithi129.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-03-16T09:29:38.828 INFO:teuthology.orchestra.run.smithi119.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-16T09:29:39.261 INFO:teuthology.orchestra.run.smithi119.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.3 MB/s | 3.1 MB 00:00 2024-03-16T09:29:39.361 INFO:teuthology.orchestra.run.smithi119.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 3.4 MB/s | 1.8 MB 00:00 2024-03-16T09:29:39.375 INFO:teuthology.orchestra.run.smithi129.stdout:(97/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 6.6 MB/s | 48 MB 00:07 2024-03-16T09:29:39.462 INFO:teuthology.orchestra.run.smithi119.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.4 MB/s | 248 kB 00:00 2024-03-16T09:29:39.504 INFO:teuthology.orchestra.run.smithi119.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 1.2 MB/s | 49 kB 00:00 2024-03-16T09:29:39.546 INFO:teuthology.orchestra.run.smithi119.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 395 kB/s | 16 kB 00:00 2024-03-16T09:29:39.629 INFO:teuthology.orchestra.run.smithi119.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.1 MB/s | 264 kB 00:00 2024-03-16T09:29:39.641 INFO:teuthology.orchestra.run.smithi077.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 4.1 MB/s | 21 MB 00:05 2024-03-16T09:29:39.708 INFO:teuthology.orchestra.run.smithi077.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 379 kB/s | 25 kB 00:00 2024-03-16T09:29:39.805 INFO:teuthology.orchestra.run.smithi119.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-16T09:29:39.880 INFO:teuthology.orchestra.run.smithi119.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 244 kB/s | 18 kB 00:00 2024-03-16T09:29:40.014 INFO:teuthology.orchestra.run.smithi119.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 662 kB 00:00 2024-03-16T09:29:40.064 INFO:teuthology.orchestra.run.smithi119.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-16T09:29:40.098 INFO:teuthology.orchestra.run.smithi119.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-16T09:29:40.110 INFO:teuthology.orchestra.run.smithi129.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 181 kB/s | 245 kB 00:01 2024-03-16T09:29:40.140 INFO:teuthology.orchestra.run.smithi119.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-16T09:29:40.173 INFO:teuthology.orchestra.run.smithi119.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 948 kB/s | 32 kB 00:00 2024-03-16T09:29:40.274 INFO:teuthology.orchestra.run.smithi119.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-16T09:29:40.307 INFO:teuthology.orchestra.run.smithi119.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 516 kB/s | 17 kB 00:00 2024-03-16T09:29:40.318 INFO:teuthology.orchestra.run.smithi129.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 269 kB/s | 248 kB 00:00 2024-03-16T09:29:40.341 INFO:teuthology.orchestra.run.smithi119.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-16T09:29:40.366 INFO:teuthology.orchestra.run.smithi119.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 6.3 MB/s | 159 kB 00:00 2024-03-16T09:29:40.392 INFO:teuthology.orchestra.run.smithi119.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-16T09:29:40.433 INFO:teuthology.orchestra.run.smithi119.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 661 kB 00:00 2024-03-16T09:29:40.517 INFO:teuthology.orchestra.run.smithi119.stdout:(49/174): ceph-mgr-diskprediction-local-19.0.0- 5.9 MB/s | 7.4 MB 00:01 2024-03-16T09:29:40.542 INFO:teuthology.orchestra.run.smithi119.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 578 kB 00:00 2024-03-16T09:29:40.568 INFO:teuthology.orchestra.run.smithi119.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 937 kB/s | 46 kB 00:00 2024-03-16T09:29:40.593 INFO:teuthology.orchestra.run.smithi119.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-16T09:29:40.618 INFO:teuthology.orchestra.run.smithi119.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-16T09:29:40.644 INFO:teuthology.orchestra.run.smithi119.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-16T09:29:40.650 INFO:teuthology.orchestra.run.smithi077.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 4.7 MB/s | 9.8 MB 00:02 2024-03-16T09:29:40.669 INFO:teuthology.orchestra.run.smithi119.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-16T09:29:40.717 INFO:teuthology.orchestra.run.smithi077.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 3.3 MB/s | 221 kB 00:00 2024-03-16T09:29:40.767 INFO:teuthology.orchestra.run.smithi077.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 543 kB/s | 27 kB 00:00 2024-03-16T09:29:40.769 INFO:teuthology.orchestra.run.smithi119.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-16T09:29:40.811 INFO:teuthology.orchestra.run.smithi119.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-16T09:29:40.817 INFO:teuthology.orchestra.run.smithi077.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 634 kB/s | 32 kB 00:00 2024-03-16T09:29:40.845 INFO:teuthology.orchestra.run.smithi119.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-16T09:29:40.870 INFO:teuthology.orchestra.run.smithi119.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-16T09:29:40.944 INFO:teuthology.orchestra.run.smithi129.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 56 kB/s | 35 kB 00:00 2024-03-16T09:29:40.979 INFO:teuthology.orchestra.run.smithi119.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 43 MB/s | 5.7 MB 00:00 2024-03-16T09:29:41.010 INFO:teuthology.orchestra.run.smithi129.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 241 kB/s | 216 kB 00:00 2024-03-16T09:29:41.012 INFO:teuthology.orchestra.run.smithi119.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.2 MB/s | 1.0 MB 00:00 2024-03-16T09:29:41.071 INFO:teuthology.orchestra.run.smithi119.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 860 kB/s | 78 kB 00:00 2024-03-16T09:29:41.093 INFO:teuthology.orchestra.run.smithi077.stdout:(14/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 4.4 MB/s | 17 MB 00:03 2024-03-16T09:29:41.126 INFO:teuthology.orchestra.run.smithi077.stdout:(15/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 2.4 MB/s | 747 kB 00:00 2024-03-16T09:29:41.151 INFO:teuthology.orchestra.run.smithi077.stdout:(16/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.7 MB/s | 163 kB 00:00 2024-03-16T09:29:41.176 INFO:teuthology.orchestra.run.smithi077.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 2.5 MB/s | 127 kB 00:00 2024-03-16T09:29:41.179 INFO:teuthology.orchestra.run.smithi119.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-16T09:29:41.205 INFO:teuthology.orchestra.run.smithi119.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 185 kB/s | 24 kB 00:00 2024-03-16T09:29:41.230 INFO:teuthology.orchestra.run.smithi119.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-16T09:29:41.252 INFO:teuthology.orchestra.run.smithi077.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 4.9 MB/s | 501 kB 00:00 2024-03-16T09:29:41.264 INFO:teuthology.orchestra.run.smithi119.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-16T09:29:41.289 INFO:teuthology.orchestra.run.smithi119.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 540 kB/s | 32 kB 00:00 2024-03-16T09:29:41.310 INFO:teuthology.orchestra.run.smithi077.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 776 kB/s | 45 kB 00:00 2024-03-16T09:29:41.369 INFO:teuthology.orchestra.run.smithi077.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 2.2 MB/s | 131 kB 00:00 2024-03-16T09:29:41.390 INFO:teuthology.orchestra.run.smithi119.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-16T09:29:41.415 INFO:teuthology.orchestra.run.smithi119.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-16T09:29:41.440 INFO:teuthology.orchestra.run.smithi119.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-16T09:29:41.444 INFO:teuthology.orchestra.run.smithi077.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.1 MB/s | 162 kB 00:00 2024-03-16T09:29:41.457 INFO:teuthology.orchestra.run.smithi119.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-16T09:29:41.482 INFO:teuthology.orchestra.run.smithi119.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 985 kB/s | 41 kB 00:00 2024-03-16T09:29:41.508 INFO:teuthology.orchestra.run.smithi119.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-16T09:29:41.541 INFO:teuthology.orchestra.run.smithi119.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-16T09:29:41.553 INFO:teuthology.orchestra.run.smithi077.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.9 MB/s | 320 kB 00:00 2024-03-16T09:29:41.567 INFO:teuthology.orchestra.run.smithi119.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-16T09:29:41.584 INFO:teuthology.orchestra.run.smithi119.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-16T09:29:41.594 INFO:teuthology.orchestra.run.smithi129.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 68 kB/s | 44 kB 00:00 2024-03-16T09:29:41.600 INFO:teuthology.orchestra.run.smithi119.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-16T09:29:41.662 INFO:teuthology.orchestra.run.smithi077.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.7 MB/s | 298 kB 00:00 2024-03-16T09:29:41.676 INFO:teuthology.orchestra.run.smithi119.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 523 kB/s | 39 kB 00:00 2024-03-16T09:29:41.711 INFO:teuthology.orchestra.run.smithi129.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 134 kB/s | 94 kB 00:00 2024-03-16T09:29:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 2.0 MB/s | 99 kB 00:00 2024-03-16T09:29:41.771 INFO:teuthology.orchestra.run.smithi077.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-16T09:29:41.811 INFO:teuthology.orchestra.run.smithi129.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-16T09:29:41.909 INFO:teuthology.orchestra.run.smithi119.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-16T09:29:41.926 INFO:teuthology.orchestra.run.smithi119.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 165 kB/s | 41 kB 00:00 2024-03-16T09:29:41.937 INFO:teuthology.orchestra.run.smithi077.stdout:(26/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 6.8 MB/s | 5.1 MB 00:00 2024-03-16T09:29:41.951 INFO:teuthology.orchestra.run.smithi119.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-16T09:29:41.968 INFO:teuthology.orchestra.run.smithi119.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-16T09:29:41.970 INFO:teuthology.orchestra.run.smithi129.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-03-16T09:29:41.985 INFO:teuthology.orchestra.run.smithi119.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-16T09:29:42.002 INFO:teuthology.orchestra.run.smithi119.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-16T09:29:42.004 INFO:teuthology.orchestra.run.smithi077.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-16T09:29:42.028 INFO:teuthology.orchestra.run.smithi119.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-16T09:29:42.053 INFO:teuthology.orchestra.run.smithi119.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-16T09:29:42.055 INFO:teuthology.orchestra.run.smithi077.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 678 kB/s | 34 kB 00:00 2024-03-16T09:29:42.087 INFO:teuthology.orchestra.run.smithi119.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-16T09:29:42.112 INFO:teuthology.orchestra.run.smithi119.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-16T09:29:42.113 INFO:teuthology.orchestra.run.smithi077.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.3 MB/s | 139 kB 00:00 2024-03-16T09:29:42.137 INFO:teuthology.orchestra.run.smithi119.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-16T09:29:42.162 INFO:teuthology.orchestra.run.smithi119.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-16T09:29:42.278 INFO:teuthology.orchestra.run.smithi129.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-16T09:29:42.305 INFO:teuthology.orchestra.run.smithi077.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 5.8 MB/s | 3.1 MB 00:00 2024-03-16T09:29:42.329 INFO:teuthology.orchestra.run.smithi119.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-16T09:29:42.354 INFO:teuthology.orchestra.run.smithi129.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 260 kB/s | 19 kB 00:00 2024-03-16T09:29:42.387 INFO:teuthology.orchestra.run.smithi129.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 740 kB/s | 309 kB 00:00 2024-03-16T09:29:42.413 INFO:teuthology.orchestra.run.smithi119.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-16T09:29:42.431 INFO:teuthology.orchestra.run.smithi077.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 5.7 MB/s | 1.8 MB 00:00 2024-03-16T09:29:42.455 INFO:teuthology.orchestra.run.smithi119.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-03-16T09:29:42.462 INFO:teuthology.orchestra.run.smithi129.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-16T09:29:42.480 INFO:teuthology.orchestra.run.smithi119.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-03-16T09:29:42.481 INFO:teuthology.orchestra.run.smithi077.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 4.9 MB/s | 248 kB 00:00 2024-03-16T09:29:42.514 INFO:teuthology.orchestra.run.smithi119.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 364 kB/s | 21 kB 00:00 2024-03-16T09:29:42.538 INFO:teuthology.orchestra.run.smithi129.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-16T09:29:42.540 INFO:teuthology.orchestra.run.smithi077.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 848 kB/s | 49 kB 00:00 2024-03-16T09:29:42.540 INFO:teuthology.orchestra.run.smithi119.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 430 kB/s | 25 kB 00:00 2024-03-16T09:29:42.598 INFO:teuthology.orchestra.run.smithi077.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 282 kB/s | 16 kB 00:00 2024-03-16T09:29:42.640 INFO:teuthology.orchestra.run.smithi119.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-16T09:29:42.646 INFO:teuthology.orchestra.run.smithi129.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-16T09:29:42.665 INFO:teuthology.orchestra.run.smithi077.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.9 MB/s | 264 kB 00:00 2024-03-16T09:29:42.671 INFO:teuthology.orchestra.run.smithi129.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-03-16T09:29:42.722 INFO:teuthology.orchestra.run.smithi129.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-16T09:29:42.723 INFO:teuthology.orchestra.run.smithi119.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-03-16T09:29:42.806 INFO:teuthology.orchestra.run.smithi129.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 6.5 MB/s | 548 kB 00:00 2024-03-16T09:29:42.807 INFO:teuthology.orchestra.run.smithi119.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 216 kB 00:00 2024-03-16T09:29:42.832 INFO:teuthology.orchestra.run.smithi077.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-16T09:29:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-16T09:29:42.857 INFO:teuthology.orchestra.run.smithi119.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 700 kB/s | 35 kB 00:00 2024-03-16T09:29:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 578 kB/s | 43 kB 00:00 2024-03-16T09:29:42.890 INFO:teuthology.orchestra.run.smithi077.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 314 kB/s | 18 kB 00:00 2024-03-16T09:29:42.909 INFO:teuthology.orchestra.run.smithi119.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-16T09:29:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-16T09:29:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-16T09:29:42.966 INFO:teuthology.orchestra.run.smithi119.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-16T09:29:42.999 INFO:teuthology.orchestra.run.smithi129.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-16T09:29:43.024 INFO:teuthology.orchestra.run.smithi077.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-16T09:29:43.032 INFO:teuthology.orchestra.run.smithi129.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-16T09:29:43.050 INFO:teuthology.orchestra.run.smithi119.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-16T09:29:43.074 INFO:teuthology.orchestra.run.smithi077.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-16T09:29:43.082 INFO:teuthology.orchestra.run.smithi129.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-16T09:29:43.108 INFO:teuthology.orchestra.run.smithi129.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 470 kB/s | 35 kB 00:00 2024-03-16T09:29:43.108 INFO:teuthology.orchestra.run.smithi077.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-16T09:29:43.150 INFO:teuthology.orchestra.run.smithi077.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 307 kB 00:00 2024-03-16T09:29:43.166 INFO:teuthology.orchestra.run.smithi129.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-16T09:29:43.183 INFO:teuthology.orchestra.run.smithi077.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 951 kB/s | 32 kB 00:00 2024-03-16T09:29:43.191 INFO:teuthology.orchestra.run.smithi119.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-03-16T09:29:43.225 INFO:teuthology.orchestra.run.smithi129.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-03-16T09:29:43.267 INFO:teuthology.orchestra.run.smithi077.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 36 MB/s | 3.0 MB 00:00 2024-03-16T09:29:43.279 INFO:teuthology.orchestra.run.smithi129.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-16T09:29:43.300 INFO:teuthology.orchestra.run.smithi077.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 517 kB/s | 17 kB 00:00 2024-03-16T09:29:43.309 INFO:teuthology.orchestra.run.smithi129.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-16T09:29:43.351 INFO:teuthology.orchestra.run.smithi129.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 248 kB/s | 18 kB 00:00 2024-03-16T09:29:43.376 INFO:teuthology.orchestra.run.smithi129.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-03-16T09:29:43.400 INFO:teuthology.orchestra.run.smithi119.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.7 MB/s | 4.1 MB 00:00 2024-03-16T09:29:43.410 INFO:teuthology.orchestra.run.smithi077.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 93 kB/s | 10 kB 00:00 2024-03-16T09:29:43.418 INFO:teuthology.orchestra.run.smithi129.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 291 kB/s | 19 kB 00:00 2024-03-16T09:29:43.443 INFO:teuthology.orchestra.run.smithi129.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-03-16T09:29:43.493 INFO:teuthology.orchestra.run.smithi077.stdout:(46/174): ceph-mgr-diskprediction-local-19.0.0- 6.2 MB/s | 7.4 MB 00:01 2024-03-16T09:29:43.502 INFO:teuthology.orchestra.run.smithi129.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-16T09:29:43.518 INFO:teuthology.orchestra.run.smithi077.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-16T09:29:43.529 INFO:teuthology.orchestra.run.smithi129.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 553 kB/s | 46 kB 00:00 2024-03-16T09:29:43.544 INFO:teuthology.orchestra.run.smithi077.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-16T09:29:43.569 INFO:teuthology.orchestra.run.smithi077.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T09:29:43.577 INFO:teuthology.orchestra.run.smithi129.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-16T09:29:43.594 INFO:teuthology.orchestra.run.smithi129.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 864 kB/s | 58 kB 00:00 2024-03-16T09:29:43.603 INFO:teuthology.orchestra.run.smithi077.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-16T09:29:43.619 INFO:teuthology.orchestra.run.smithi077.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-03-16T09:29:43.645 INFO:teuthology.orchestra.run.smithi077.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-16T09:29:43.670 INFO:teuthology.orchestra.run.smithi077.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-16T09:29:43.695 INFO:teuthology.orchestra.run.smithi077.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-16T09:29:43.712 INFO:teuthology.orchestra.run.smithi077.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 828 kB/s | 34 kB 00:00 2024-03-16T09:29:43.778 INFO:teuthology.orchestra.run.smithi129.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-16T09:29:43.795 INFO:teuthology.orchestra.run.smithi129.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 82 kB/s | 16 kB 00:00 2024-03-16T09:29:43.821 INFO:teuthology.orchestra.run.smithi077.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-16T09:29:43.854 INFO:teuthology.orchestra.run.smithi129.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-16T09:29:43.854 INFO:teuthology.orchestra.run.smithi077.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-16T09:29:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-16T09:29:43.905 INFO:teuthology.orchestra.run.smithi077.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.8 MB/s | 811 kB 00:00 2024-03-16T09:29:43.921 INFO:teuthology.orchestra.run.smithi129.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-03-16T09:29:43.930 INFO:teuthology.orchestra.run.smithi077.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.8 MB/s | 134 kB 00:00 2024-03-16T09:29:43.946 INFO:teuthology.orchestra.run.smithi129.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-16T09:29:43.996 INFO:teuthology.orchestra.run.smithi129.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 789 kB/s | 59 kB 00:00 2024-03-16T09:29:44.022 INFO:teuthology.orchestra.run.smithi129.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-03-16T09:29:44.033 INFO:teuthology.orchestra.run.smithi119.stdout:(106/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 5.1 MB/s | 48 MB 00:09 2024-03-16T09:29:44.056 INFO:teuthology.orchestra.run.smithi077.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-16T09:29:44.080 INFO:teuthology.orchestra.run.smithi129.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-03-16T09:29:44.089 INFO:teuthology.orchestra.run.smithi077.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-16T09:29:44.106 INFO:teuthology.orchestra.run.smithi129.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-16T09:29:44.114 INFO:teuthology.orchestra.run.smithi077.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-16T09:29:44.156 INFO:teuthology.orchestra.run.smithi129.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-16T09:29:44.198 INFO:teuthology.orchestra.run.smithi129.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-16T09:29:44.232 INFO:teuthology.orchestra.run.smithi129.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 267 kB/s | 20 kB 00:00 2024-03-16T09:29:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-16T09:29:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-16T09:29:44.273 INFO:teuthology.orchestra.run.smithi129.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-16T09:29:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 362 kB/s | 18 kB 00:00 2024-03-16T09:29:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-16T09:29:44.326 INFO:teuthology.orchestra.run.smithi119.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 66 kB/s | 19 kB 00:00 2024-03-16T09:29:44.332 INFO:teuthology.orchestra.run.smithi129.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-16T09:29:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-16T09:29:44.493 INFO:teuthology.orchestra.run.smithi119.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 421 kB/s | 548 kB 00:01 2024-03-16T09:29:44.518 INFO:teuthology.orchestra.run.smithi119.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 276 kB/s | 309 kB 00:01 2024-03-16T09:29:44.769 INFO:teuthology.orchestra.run.smithi119.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-03-16T09:29:44.791 INFO:teuthology.orchestra.run.smithi077.stdout:(68/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 9.4 MB/s | 48 MB 00:05 2024-03-16T09:29:44.816 INFO:teuthology.orchestra.run.smithi129.stdout:(150/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-03-16T09:29:44.850 INFO:teuthology.orchestra.run.smithi077.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.8 MB/s | 4.8 MB 00:00 2024-03-16T09:29:44.857 INFO:teuthology.orchestra.run.smithi129.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-16T09:29:44.875 INFO:teuthology.orchestra.run.smithi077.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 403 kB/s | 217 kB 00:00 2024-03-16T09:29:44.885 INFO:teuthology.orchestra.run.smithi119.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 132 kB/s | 49 kB 00:00 2024-03-16T09:29:44.900 INFO:teuthology.orchestra.run.smithi077.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 174 kB 00:00 2024-03-16T09:29:44.916 INFO:teuthology.orchestra.run.smithi129.stdout:(152/174): openssl-devel-3.0.7-27.el9.x86_64.rp 684 kB/s | 4.1 MB 00:06 2024-03-16T09:29:44.925 INFO:teuthology.orchestra.run.smithi077.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-03-16T09:29:44.950 INFO:teuthology.orchestra.run.smithi077.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 552 kB/s | 41 kB 00:00 2024-03-16T09:29:44.976 INFO:teuthology.orchestra.run.smithi077.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-16T09:29:45.001 INFO:teuthology.orchestra.run.smithi077.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-16T09:29:45.026 INFO:teuthology.orchestra.run.smithi077.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-16T09:29:45.041 INFO:teuthology.orchestra.run.smithi129.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-16T09:29:45.051 INFO:teuthology.orchestra.run.smithi077.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 636 kB/s | 47 kB 00:00 2024-03-16T09:29:45.077 INFO:teuthology.orchestra.run.smithi077.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 857 kB/s | 43 kB 00:00 2024-03-16T09:29:45.092 INFO:teuthology.orchestra.run.smithi129.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 10 MB/s | 1.8 MB 00:00 2024-03-16T09:29:45.117 INFO:teuthology.orchestra.run.smithi129.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 563 kB/s | 42 kB 00:00 2024-03-16T09:29:45.127 INFO:teuthology.orchestra.run.smithi119.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 188 kB/s | 67 kB 00:00 2024-03-16T09:29:45.167 INFO:teuthology.orchestra.run.smithi129.stdout:(156/174): librados2-19.0.0-2162.g3d78f5ba.el9. 3.8 MB/s | 3.4 MB 00:00 2024-03-16T09:29:45.192 INFO:teuthology.orchestra.run.smithi129.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 949 kB/s | 71 kB 00:00 2024-03-16T09:29:45.202 INFO:teuthology.orchestra.run.smithi077.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 263 kB/s | 39 kB 00:00 2024-03-16T09:29:45.218 INFO:teuthology.orchestra.run.smithi129.stdout:(158/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-16T09:29:45.243 INFO:teuthology.orchestra.run.smithi129.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-16T09:29:45.268 INFO:teuthology.orchestra.run.smithi129.stdout:(160/174): elfutils-debuginfod-client-0.190-2.e 362 kB/s | 36 kB 00:00 2024-03-16T09:29:45.294 INFO:teuthology.orchestra.run.smithi129.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-16T09:29:45.319 INFO:teuthology.orchestra.run.smithi129.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-03-16T09:29:45.336 INFO:teuthology.orchestra.run.smithi129.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-16T09:29:45.344 INFO:teuthology.orchestra.run.smithi119.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 329 kB/s | 151 kB 00:00 2024-03-16T09:29:45.361 INFO:teuthology.orchestra.run.smithi077.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-16T09:29:45.361 INFO:teuthology.orchestra.run.smithi129.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-16T09:29:45.385 INFO:teuthology.orchestra.run.smithi077.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 134 kB/s | 41 kB 00:00 2024-03-16T09:29:45.386 INFO:teuthology.orchestra.run.smithi129.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-16T09:29:45.411 INFO:teuthology.orchestra.run.smithi077.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-16T09:29:45.411 INFO:teuthology.orchestra.run.smithi129.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 574 kB/s | 29 kB 00:00 2024-03-16T09:29:45.436 INFO:teuthology.orchestra.run.smithi077.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-16T09:29:45.437 INFO:teuthology.orchestra.run.smithi129.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-16T09:29:45.461 INFO:teuthology.orchestra.run.smithi077.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-16T09:29:45.486 INFO:teuthology.orchestra.run.smithi077.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-16T09:29:45.495 INFO:teuthology.orchestra.run.smithi129.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-16T09:29:45.511 INFO:teuthology.orchestra.run.smithi119.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-03-16T09:29:45.512 INFO:teuthology.orchestra.run.smithi077.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-16T09:29:45.537 INFO:teuthology.orchestra.run.smithi077.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-16T09:29:45.545 INFO:teuthology.orchestra.run.smithi129.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 7.6 MB/s | 1.2 MB 00:00 2024-03-16T09:29:45.570 INFO:teuthology.orchestra.run.smithi077.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-16T09:29:45.571 INFO:teuthology.orchestra.run.smithi129.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.0 MB/s | 465 kB 00:00 2024-03-16T09:29:45.588 INFO:teuthology.orchestra.run.smithi129.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 744 kB/s | 31 kB 00:00 2024-03-16T09:29:45.596 INFO:teuthology.orchestra.run.smithi077.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-16T09:29:45.621 INFO:teuthology.orchestra.run.smithi077.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-16T09:29:45.638 INFO:teuthology.orchestra.run.smithi129.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-16T09:29:45.646 INFO:teuthology.orchestra.run.smithi077.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-16T09:29:45.680 INFO:teuthology.orchestra.run.smithi129.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-16T09:29:45.803 INFO:teuthology.orchestra.run.smithi119.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-16T09:29:45.895 INFO:teuthology.orchestra.run.smithi119.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 113 kB/s | 43 kB 00:00 2024-03-16T09:29:45.995 INFO:teuthology.orchestra.run.smithi119.stdout:(117/174): python3-asyncssh-2.13.2-4.el9.noarch 842 kB/s | 548 kB 00:00 2024-03-16T09:29:46.087 INFO:teuthology.orchestra.run.smithi119.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 113 kB/s | 32 kB 00:00 2024-03-16T09:29:46.163 INFO:teuthology.orchestra.run.smithi077.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 346 kB/s | 187 kB 00:00 2024-03-16T09:29:46.189 INFO:teuthology.orchestra.run.smithi077.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 360 kB/s | 195 kB 00:00 2024-03-16T09:29:46.221 INFO:teuthology.orchestra.run.smithi119.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 46 kB/s | 15 kB 00:00 2024-03-16T09:29:46.272 INFO:teuthology.orchestra.run.smithi077.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-16T09:29:46.288 INFO:teuthology.orchestra.run.smithi129.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 106 kB/s | 74 kB 00:00 2024-03-16T09:29:46.297 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:29:46.297 INFO:teuthology.orchestra.run.smithi129.stdout:Total 13 MB/s | 224 MB 00:17 2024-03-16T09:29:46.389 INFO:teuthology.orchestra.run.smithi077.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 107 kB/s | 21 kB 00:00 2024-03-16T09:29:46.423 INFO:teuthology.orchestra.run.smithi077.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-03-16T09:29:46.448 INFO:teuthology.orchestra.run.smithi077.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 170 kB/s | 25 kB 00:00 2024-03-16T09:29:46.463 INFO:teuthology.orchestra.run.smithi119.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 368 kB/s | 172 kB 00:00 2024-03-16T09:29:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-16T09:29:46.615 INFO:teuthology.orchestra.run.smithi077.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.9 MB/s | 216 kB 00:00 2024-03-16T09:29:46.680 INFO:teuthology.orchestra.run.smithi119.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 606 kB/s | 358 kB 00:00 2024-03-16T09:29:46.690 INFO:teuthology.orchestra.run.smithi077.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 14 MB/s | 4.1 MB 00:00 2024-03-16T09:29:46.724 INFO:teuthology.orchestra.run.smithi077.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 322 kB/s | 35 kB 00:00 2024-03-16T09:29:46.747 INFO:teuthology.orchestra.run.smithi119.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 124 kB/s | 35 kB 00:00 2024-03-16T09:29:46.749 INFO:teuthology.orchestra.run.smithi077.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 824 kB/s | 248 kB 00:00 2024-03-16T09:29:46.766 INFO:teuthology.orchestra.run.smithi077.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 591 kB/s | 44 kB 00:00 2024-03-16T09:29:46.781 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-16T09:29:46.791 INFO:teuthology.orchestra.run.smithi077.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-16T09:29:46.797 INFO:teuthology.orchestra.run.smithi119.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 526 kB/s | 303 kB 00:00 2024-03-16T09:29:46.908 INFO:teuthology.orchestra.run.smithi077.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-03-16T09:29:47.084 INFO:teuthology.orchestra.run.smithi077.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-03-16T09:29:47.090 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-16T09:29:47.090 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-16T09:29:47.201 INFO:teuthology.orchestra.run.smithi077.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-16T09:29:47.259 INFO:teuthology.orchestra.run.smithi077.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-03-16T09:29:47.264 INFO:teuthology.orchestra.run.smithi119.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 310 kB/s | 144 kB 00:00 2024-03-16T09:29:47.289 INFO:teuthology.orchestra.run.smithi119.stdout:(125/174): python3-google-auth-2.28.0-1.el9.noa 333 kB/s | 203 kB 00:00 2024-03-16T09:29:47.359 INFO:teuthology.orchestra.run.smithi077.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-16T09:29:47.384 INFO:teuthology.orchestra.run.smithi077.stdout:(110/174): gperftools-libs-2.9.1-2.el9.x86_64.r 649 kB/s | 309 kB 00:00 2024-03-16T09:29:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 53 kB/s | 11 kB 00:00 2024-03-16T09:29:47.468 INFO:teuthology.orchestra.run.smithi077.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-16T09:29:47.493 INFO:teuthology.orchestra.run.smithi077.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 363 kB/s | 49 kB 00:00 2024-03-16T09:29:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-16T09:29:47.565 INFO:teuthology.orchestra.run.smithi119.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.5 MB/s | 2.0 MB 00:00 2024-03-16T09:29:47.577 INFO:teuthology.orchestra.run.smithi077.stdout:(114/174): libunwind-1.6.2-1.el9.x86_64.rpm 350 kB/s | 67 kB 00:00 2024-03-16T09:29:47.590 INFO:teuthology.orchestra.run.smithi119.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 59 kB/s | 18 kB 00:00 2024-03-16T09:29:47.636 INFO:teuthology.orchestra.run.smithi077.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-16T09:29:47.670 INFO:teuthology.orchestra.run.smithi077.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-16T09:29:47.694 INFO:teuthology.orchestra.run.smithi077.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-03-16T09:29:47.720 INFO:teuthology.orchestra.run.smithi077.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 384 kB/s | 32 kB 00:00 2024-03-16T09:29:47.765 INFO:teuthology.orchestra.run.smithi119.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 77 kB/s | 23 kB 00:00 2024-03-16T09:29:47.828 INFO:teuthology.orchestra.run.smithi077.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 94 kB/s | 15 kB 00:00 2024-03-16T09:29:47.849 INFO:teuthology.orchestra.run.smithi119.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 68 kB/s | 19 kB 00:00 2024-03-16T09:29:47.854 INFO:teuthology.orchestra.run.smithi077.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-03-16T09:29:47.874 INFO:teuthology.orchestra.run.smithi119.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 69 kB/s | 20 kB 00:00 2024-03-16T09:29:47.879 INFO:teuthology.orchestra.run.smithi077.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-03-16T09:29:47.921 INFO:teuthology.orchestra.run.smithi077.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-16T09:29:47.946 INFO:teuthology.orchestra.run.smithi077.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-03-16T09:29:47.972 INFO:teuthology.orchestra.run.smithi077.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.1 MB/s | 203 kB 00:00 2024-03-16T09:29:48.039 INFO:teuthology.orchestra.run.smithi077.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-03-16T09:29:48.064 INFO:teuthology.orchestra.run.smithi077.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-16T09:29:48.114 INFO:teuthology.orchestra.run.smithi077.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-16T09:29:48.139 INFO:teuthology.orchestra.run.smithi077.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-16T09:29:48.165 INFO:teuthology.orchestra.run.smithi077.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-03-16T09:29:48.206 INFO:teuthology.orchestra.run.smithi077.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-16T09:29:48.231 INFO:teuthology.orchestra.run.smithi077.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-16T09:29:48.232 INFO:teuthology.orchestra.run.smithi119.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 121 kB/s | 46 kB 00:00 2024-03-16T09:29:48.258 INFO:teuthology.orchestra.run.smithi119.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 205 kB/s | 79 kB 00:00 2024-03-16T09:29:48.323 INFO:teuthology.orchestra.run.smithi077.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-03-16T09:29:48.357 INFO:teuthology.orchestra.run.smithi077.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 629 kB/s | 79 kB 00:00 2024-03-16T09:29:48.432 INFO:teuthology.orchestra.run.smithi077.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 532 kB/s | 58 kB 00:00 2024-03-16T09:29:48.524 INFO:teuthology.orchestra.run.smithi077.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 2.8 MB/s | 1.0 MB 00:00 2024-03-16T09:29:48.525 INFO:teuthology.orchestra.run.smithi119.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 1.3 MB/s | 1.0 MB 00:00 2024-03-16T09:29:48.541 INFO:teuthology.orchestra.run.smithi077.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-16T09:29:48.566 INFO:teuthology.orchestra.run.smithi077.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-16T09:29:48.609 INFO:teuthology.orchestra.run.smithi077.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-16T09:29:48.625 INFO:teuthology.orchestra.run.smithi119.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 147 kB/s | 58 kB 00:00 2024-03-16T09:29:48.626 INFO:teuthology.orchestra.run.smithi077.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-03-16T09:29:48.642 INFO:teuthology.orchestra.run.smithi077.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-16T09:29:48.709 INFO:teuthology.orchestra.run.smithi077.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-16T09:29:48.726 INFO:teuthology.orchestra.run.smithi077.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 427 kB/s | 36 kB 00:00 2024-03-16T09:29:48.743 INFO:teuthology.orchestra.run.smithi077.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-03-16T09:29:48.785 INFO:teuthology.orchestra.run.smithi077.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-16T09:29:48.825 INFO:teuthology.orchestra.run.smithi119.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 55 kB/s | 16 kB 00:00 2024-03-16T09:29:48.835 INFO:teuthology.orchestra.run.smithi077.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 975 kB/s | 90 kB 00:00 2024-03-16T09:29:48.861 INFO:teuthology.orchestra.run.smithi077.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-16T09:29:48.892 INFO:teuthology.orchestra.run.smithi119.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 430 kB/s | 272 kB 00:00 2024-03-16T09:29:48.961 INFO:teuthology.orchestra.run.smithi077.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-03-16T09:29:48.986 INFO:teuthology.orchestra.run.smithi077.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-16T09:29:49.053 INFO:teuthology.orchestra.run.smithi077.stdout:(149/174): python3-zc-lockfile-2.0-10.el9.noarc 92 kB/s | 20 kB 00:00 2024-03-16T09:29:49.084 INFO:teuthology.orchestra.run.smithi119.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 196 kB/s | 90 kB 00:00 2024-03-16T09:29:49.101 INFO:teuthology.orchestra.run.smithi119.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 102 kB/s | 28 kB 00:00 2024-03-16T09:29:49.170 INFO:teuthology.orchestra.run.smithi077.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-03-16T09:29:49.176 INFO:teuthology.orchestra.run.smithi119.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 108 kB/s | 31 kB 00:00 2024-03-16T09:29:49.219 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-16T09:29:49.220 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-16T09:29:49.295 INFO:teuthology.orchestra.run.smithi077.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-16T09:29:49.495 INFO:teuthology.orchestra.run.smithi077.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 6.7 MB/s | 3.4 MB 00:00 2024-03-16T09:29:49.602 INFO:teuthology.orchestra.run.smithi119.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 84 kB/s | 36 kB 00:00 2024-03-16T09:29:49.627 INFO:teuthology.orchestra.run.smithi119.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 112 kB/s | 59 kB 00:00 2024-03-16T09:29:49.652 INFO:teuthology.orchestra.run.smithi119.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 332 kB/s | 188 kB 00:00 2024-03-16T09:29:49.796 INFO:teuthology.orchestra.run.smithi077.stdout:(153/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-03-16T09:29:49.986 INFO:teuthology.orchestra.run.smithi119.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 133 kB/s | 51 kB 00:00 2024-03-16T09:29:50.128 INFO:teuthology.orchestra.run.smithi119.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 189 kB/s | 90 kB 00:00 2024-03-16T09:29:50.146 INFO:teuthology.orchestra.run.smithi077.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 121 kB/s | 42 kB 00:00 2024-03-16T09:29:50.170 INFO:teuthology.orchestra.run.smithi119.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 423 kB/s | 230 kB 00:00 2024-03-16T09:29:50.338 INFO:teuthology.orchestra.run.smithi077.stdout:(155/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.2 MB/s | 2.3 MB 00:01 2024-03-16T09:29:50.420 INFO:teuthology.orchestra.run.smithi119.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 68 kB/s | 20 kB 00:00 2024-03-16T09:29:50.455 INFO:teuthology.orchestra.run.smithi077.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 1.9 MB/s | 1.8 MB 00:00 2024-03-16T09:29:50.480 INFO:teuthology.orchestra.run.smithi077.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 502 kB/s | 71 kB 00:00 2024-03-16T09:29:50.505 INFO:teuthology.orchestra.run.smithi077.stdout:(158/174): NetworkManager-tui-1.46.0-1.el9.x86_ 691 kB/s | 248 kB 00:00 2024-03-16T09:29:50.556 INFO:teuthology.orchestra.run.smithi077.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 363 kB/s | 36 kB 00:00 2024-03-16T09:29:50.589 INFO:teuthology.orchestra.run.smithi077.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-16T09:29:50.612 INFO:teuthology.orchestra.run.smithi119.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 683 kB/s | 427 kB 00:00 2024-03-16T09:29:50.637 INFO:teuthology.orchestra.run.smithi119.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 410 kB/s | 192 kB 00:00 2024-03-16T09:29:50.656 INFO:teuthology.orchestra.run.smithi077.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-16T09:29:50.690 INFO:teuthology.orchestra.run.smithi077.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.4 MB/s | 256 kB 00:00 2024-03-16T09:29:50.715 INFO:teuthology.orchestra.run.smithi077.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 864 kB/s | 108 kB 00:00 2024-03-16T09:29:50.765 INFO:teuthology.orchestra.run.smithi077.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-16T09:29:50.799 INFO:teuthology.orchestra.run.smithi077.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-16T09:29:50.824 INFO:teuthology.orchestra.run.smithi077.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 587 kB/s | 64 kB 00:00 2024-03-16T09:29:50.866 INFO:teuthology.orchestra.run.smithi077.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 285 kB/s | 29 kB 00:00 2024-03-16T09:29:51.021 INFO:teuthology.orchestra.run.smithi119.stdout:(150/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 8.2 MB/s | 3.1 MB 00:00 2024-03-16T09:29:51.063 INFO:teuthology.orchestra.run.smithi119.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-16T09:29:51.100 INFO:teuthology.orchestra.run.smithi077.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.8 MB/s | 2.3 MB 00:00 2024-03-16T09:29:51.141 INFO:teuthology.orchestra.run.smithi077.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.8 MB/s | 2.2 MB 00:00 2024-03-16T09:29:51.200 INFO:teuthology.orchestra.run.smithi077.stdout:(170/174): openssl-3.0.7-27.el9.x86_64.rpm 3.0 MB/s | 1.2 MB 00:00 2024-03-16T09:29:51.205 INFO:teuthology.orchestra.run.smithi119.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 5.7 MB/s | 3.4 MB 00:00 2024-03-16T09:29:51.225 INFO:teuthology.orchestra.run.smithi077.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 465 kB 00:00 2024-03-16T09:29:51.238 INFO:teuthology.orchestra.run.smithi119.stdout:(153/174): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-03-16T09:29:51.251 INFO:teuthology.orchestra.run.smithi077.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 285 kB/s | 31 kB 00:00 2024-03-16T09:29:51.347 INFO:teuthology.orchestra.run.smithi119.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 392 kB/s | 42 kB 00:00 2024-03-16T09:29:51.401 INFO:teuthology.orchestra.run.smithi077.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-03-16T09:29:51.426 INFO:teuthology.orchestra.run.smithi077.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 371 kB/s | 74 kB 00:00 2024-03-16T09:29:51.431 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:29:51.431 INFO:teuthology.orchestra.run.smithi077.stdout:Total 13 MB/s | 224 MB 00:17 2024-03-16T09:29:51.472 INFO:teuthology.orchestra.run.smithi119.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-16T09:29:51.539 INFO:teuthology.orchestra.run.smithi119.stdout:(156/174): NetworkManager-1.46.0-1.el9.x86_64.r 4.8 MB/s | 2.3 MB 00:00 2024-03-16T09:29:51.557 INFO:teuthology.orchestra.run.smithi119.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 855 kB/s | 71 kB 00:00 2024-03-16T09:29:51.606 INFO:teuthology.orchestra.run.smithi119.stdout:(158/174): NetworkManager-libnm-1.46.0-1.el9.x8 4.5 MB/s | 1.8 MB 00:00 2024-03-16T09:29:51.632 INFO:teuthology.orchestra.run.smithi119.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 395 kB/s | 36 kB 00:00 2024-03-16T09:29:51.657 INFO:teuthology.orchestra.run.smithi119.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-16T09:29:51.682 INFO:teuthology.orchestra.run.smithi119.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-03-16T09:29:51.708 INFO:teuthology.orchestra.run.smithi119.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 305 kB/s | 23 kB 00:00 2024-03-16T09:29:51.733 INFO:teuthology.orchestra.run.smithi119.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-16T09:29:51.758 INFO:teuthology.orchestra.run.smithi119.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-16T09:29:51.783 INFO:teuthology.orchestra.run.smithi119.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-16T09:29:51.809 INFO:teuthology.orchestra.run.smithi119.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 846 kB/s | 64 kB 00:00 2024-03-16T09:29:51.834 INFO:teuthology.orchestra.run.smithi119.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 379 kB/s | 29 kB 00:00 2024-03-16T09:29:51.912 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-03-16T09:29:51.951 INFO:teuthology.orchestra.run.smithi119.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 1.2 MB 00:00 2024-03-16T09:29:52.035 INFO:teuthology.orchestra.run.smithi119.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.5 MB/s | 465 kB 00:00 2024-03-16T09:29:52.085 INFO:teuthology.orchestra.run.smithi119.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 623 kB/s | 31 kB 00:00 2024-03-16T09:29:52.202 INFO:teuthology.orchestra.run.smithi119.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-16T09:29:52.220 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-03-16T09:29:52.220 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-03-16T09:29:52.260 INFO:teuthology.orchestra.run.smithi119.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 74 kB 00:00 2024-03-16T09:29:52.302 INFO:teuthology.orchestra.run.smithi119.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 2.2 MB 00:00 2024-03-16T09:29:52.427 INFO:teuthology.orchestra.run.smithi119.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.9 MB/s | 2.3 MB 00:00 2024-03-16T09:29:52.432 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:29:52.432 INFO:teuthology.orchestra.run.smithi119.stdout:Total 10 MB/s | 224 MB 00:21 2024-03-16T09:29:52.894 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-16T09:29:53.202 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-16T09:29:53.202 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-16T09:29:54.313 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-03-16T09:29:54.314 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-03-16T09:29:55.281 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-16T09:29:55.281 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-16T09:29:56.170 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-16T09:29:56.265 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T09:29:56.412 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T09:29:56.642 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T09:29:56.655 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T09:29:56.702 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T09:29:56.756 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T09:29:56.794 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T09:29:56.845 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T09:29:56.890 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T09:29:57.222 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T09:29:57.273 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:29:57.378 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:29:57.400 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:29:57.499 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:29:57.539 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T09:29:57.586 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T09:29:57.772 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T09:29:57.818 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T09:29:57.846 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:29:57.952 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:29:57.980 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T09:29:58.045 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T09:29:58.076 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T09:29:58.099 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T09:29:58.115 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T09:29:58.307 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T09:29:58.339 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T09:29:58.370 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T09:29:58.555 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T09:29:58.681 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T09:29:58.741 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T09:29:58.870 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:29:58.934 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:29:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:29:59.264 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T09:29:59.368 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T09:29:59.397 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T09:29:59.432 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T09:29:59.467 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T09:29:59.542 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T09:29:59.595 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T09:29:59.634 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T09:29:59.699 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T09:29:59.734 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T09:29:59.776 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T09:29:59.810 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T09:29:59.837 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T09:29:59.905 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T09:29:59.941 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T09:29:59.982 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T09:30:00.014 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T09:30:00.083 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T09:30:00.128 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T09:30:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T09:30:00.351 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T09:30:00.405 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T09:30:00.921 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T09:30:01.001 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T09:30:01.083 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T09:30:01.129 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T09:30:01.139 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T09:30:01.172 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T09:30:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T09:30:01.236 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-03-16T09:30:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T09:30:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T09:30:01.330 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T09:30:01.374 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T09:30:01.445 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T09:30:01.481 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T09:30:01.558 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T09:30:01.731 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T09:30:01.753 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T09:30:01.808 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T09:30:01.821 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T09:30:01.840 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T09:30:01.873 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T09:30:01.874 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T09:30:01.920 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T09:30:02.012 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T09:30:02.054 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T09:30:02.117 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T09:30:02.189 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T09:30:02.221 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T09:30:02.258 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T09:30:02.266 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-16T09:30:02.266 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T09:30:02.304 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T09:30:02.325 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:30:02.349 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T09:30:02.359 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T09:30:02.392 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T09:30:02.392 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T09:30:02.434 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T09:30:02.447 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:30:02.476 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:30:02.480 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T09:30:02.490 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T09:30:02.552 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:30:02.575 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T09:30:02.587 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T09:30:02.623 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T09:30:02.629 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T09:30:02.645 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T09:30:02.687 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T09:30:02.713 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T09:30:02.767 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T09:30:02.780 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T09:30:02.800 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T09:30:02.807 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T09:30:02.820 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T09:30:02.834 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T09:30:02.861 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T09:30:02.861 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T09:30:02.879 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T09:30:02.886 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T09:30:02.889 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:30:02.911 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T09:30:02.996 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T09:30:03.000 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:30:03.023 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T09:30:03.097 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T09:30:03.136 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T09:30:03.136 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T09:30:03.159 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T09:30:03.174 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T09:30:03.211 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T09:30:03.245 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T09:30:03.260 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:30:03.294 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T09:30:03.342 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T09:30:03.366 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:30:03.392 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T09:30:03.394 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:30:03.398 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T09:30:03.427 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T09:30:03.463 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T09:30:03.483 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:30:03.524 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T09:30:03.568 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T09:30:03.583 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T09:30:03.658 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T09:30:03.731 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T09:30:03.743 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T09:30:03.781 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T09:30:03.797 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T09:30:03.825 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:30:03.920 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:30:03.943 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T09:30:03.991 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T09:30:03.999 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T09:30:04.004 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T09:30:04.038 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T09:30:04.061 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T09:30:04.077 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T09:30:04.129 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:30:04.131 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T09:30:04.163 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T09:30:04.194 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T09:30:04.222 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T09:30:04.236 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T09:30:04.256 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T09:30:04.268 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T09:30:04.295 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T09:30:04.299 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T09:30:04.309 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:30:04.327 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T09:30:04.357 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T09:30:04.432 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T09:30:04.467 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T09:30:04.471 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T09:30:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T09:30:04.534 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T09:30:04.536 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:30:04.569 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T09:30:04.602 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T09:30:04.646 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T09:30:04.658 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T09:30:04.662 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T09:30:04.739 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T09:30:04.742 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T09:30:04.756 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:30:04.780 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T09:30:04.783 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T09:30:04.807 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T09:30:04.814 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T09:30:04.825 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:30:04.842 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T09:30:04.861 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T09:30:04.916 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T09:30:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T09:30:04.970 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T09:30:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T09:30:05.017 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T09:30:05.053 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T09:30:05.082 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T09:30:05.116 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T09:30:05.160 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T09:30:05.161 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T09:30:05.193 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T09:30:05.220 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T09:30:05.247 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T09:30:05.280 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T09:30:05.289 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T09:30:05.315 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T09:30:05.325 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T09:30:05.350 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T09:30:05.372 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T09:30:05.397 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T09:30:05.422 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T09:30:05.467 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T09:30:05.469 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T09:30:05.508 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T09:30:05.511 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T09:30:05.573 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T09:30:05.585 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T09:30:05.608 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T09:30:05.650 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T09:30:05.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T09:30:05.719 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T09:30:05.728 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T09:30:05.737 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T09:30:05.772 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T09:30:05.780 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T09:30:05.797 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T09:30:05.814 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T09:30:05.832 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T09:30:05.862 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T09:30:05.881 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T09:30:05.913 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T09:30:05.941 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T09:30:05.974 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T09:30:05.982 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T09:30:06.036 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T09:30:06.110 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T09:30:06.161 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T09:30:06.194 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T09:30:06.242 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T09:30:06.273 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T09:30:06.288 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T09:30:06.288 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T09:30:06.383 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T09:30:06.475 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T09:30:06.537 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T09:30:06.546 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T09:30:06.578 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T09:30:06.618 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T09:30:06.655 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T09:30:06.678 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T09:30:06.802 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T09:30:06.875 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T09:30:06.912 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T09:30:06.953 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T09:30:06.954 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T09:30:07.004 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T09:30:07.013 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T09:30:07.024 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T09:30:07.049 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T09:30:07.093 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T09:30:07.131 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T09:30:07.161 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T09:30:07.162 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T09:30:07.254 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T09:30:07.298 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T09:30:07.435 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T09:30:07.439 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T09:30:07.486 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T09:30:07.541 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T09:30:07.614 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T09:30:07.645 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T09:30:07.652 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T09:30:07.681 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T09:30:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T09:30:07.747 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T09:30:07.764 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T09:30:07.790 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T09:30:07.807 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T09:30:07.850 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T09:30:07.934 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T09:30:07.982 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T09:30:07.984 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T09:30:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T09:30:08.039 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T09:30:08.060 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T09:30:08.110 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T09:30:08.111 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T09:30:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T09:30:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T09:30:08.204 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T09:30:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T09:30:08.235 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T09:30:08.245 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T09:30:08.262 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T09:30:08.283 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T09:30:08.306 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T09:30:08.309 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T09:30:08.313 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T09:30:08.333 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T09:30:08.344 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T09:30:08.347 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T09:30:08.375 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T09:30:08.403 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T09:30:08.434 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T09:30:08.491 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T09:30:08.536 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T09:30:08.566 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T09:30:08.566 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T09:30:08.608 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T09:30:08.674 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T09:30:08.701 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T09:30:08.717 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T09:30:08.722 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T09:30:08.742 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T09:30:08.750 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T09:30:08.765 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T09:30:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T09:30:08.807 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T09:30:08.820 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T09:30:08.832 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T09:30:08.840 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T09:30:08.871 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T09:30:08.908 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T09:30:08.921 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T09:30:09.027 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T09:30:09.048 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T09:30:09.079 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T09:30:09.147 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T09:30:09.190 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T09:30:09.205 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T09:30:09.238 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T09:30:09.244 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T09:30:09.276 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T09:30:09.294 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T09:30:09.306 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T09:30:09.335 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T09:30:09.369 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T09:30:09.370 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T09:30:09.395 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T09:30:09.408 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T09:30:09.449 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T09:30:09.449 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T09:30:09.475 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T09:30:09.478 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T09:30:09.480 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T09:30:09.519 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T09:30:09.554 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T09:30:09.588 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T09:30:09.621 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T09:30:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T09:30:09.662 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T09:30:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T09:30:09.697 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T09:30:09.724 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T09:30:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T09:30:09.783 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T09:30:09.785 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T09:30:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T09:30:09.856 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T09:30:09.866 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T09:30:09.888 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T09:30:09.911 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T09:30:09.918 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T09:30:09.951 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T09:30:09.994 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T09:30:09.998 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T09:30:10.028 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T09:30:10.062 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T09:30:10.094 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T09:30:10.102 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T09:30:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T09:30:10.146 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T09:30:10.234 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T09:30:10.316 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T09:30:10.360 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T09:30:10.392 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T09:30:10.438 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T09:30:10.536 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T09:30:10.584 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T09:30:10.863 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T09:30:10.907 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T09:30:10.941 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T09:30:10.989 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T09:30:11.068 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T09:30:11.110 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T09:30:11.285 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T09:30:11.298 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T09:30:11.323 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T09:30:11.340 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T09:30:11.374 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T09:30:11.400 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T09:30:11.422 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T09:30:11.500 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T09:30:11.533 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T09:30:11.701 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T09:30:11.739 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T09:30:11.756 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:11.810 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:11.816 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T09:30:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T09:30:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T09:30:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T09:30:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T09:30:12.483 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T09:30:12.483 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T09:30:12.483 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T09:30:12.483 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:12.524 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T09:30:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T09:30:13.357 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T09:30:13.448 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T09:30:13.479 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T09:30:13.510 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T09:30:13.538 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T09:30:13.789 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T09:30:13.850 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T09:30:13.875 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T09:30:13.880 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T09:30:13.920 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T09:30:13.951 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T09:30:13.957 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T09:30:13.979 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T09:30:13.988 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T09:30:14.025 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T09:30:14.291 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T09:30:14.317 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T09:30:14.389 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T09:30:14.422 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T09:30:14.458 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T09:30:14.485 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T09:30:14.529 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T09:30:14.591 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T09:30:14.613 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T09:30:14.918 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T09:30:14.945 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T09:30:15.016 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T09:30:15.038 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T09:30:16.869 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:16.901 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:17.287 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:17.318 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:17.463 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:17.496 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T09:30:17.586 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T09:30:17.586 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T09:30:17.586 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T09:30:17.586 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:17.623 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T09:30:17.857 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T09:30:17.888 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T09:30:17.981 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T09:30:17.981 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T09:30:17.981 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T09:30:17.981 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:18.039 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T09:30:24.470 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T09:30:24.470 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-03-16T09:30:24.470 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-03-16T09:30:24.471 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-03-16T09:30:24.471 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-03-16T09:30:24.471 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-03-16T09:30:24.471 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-03-16T09:30:24.471 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-03-16T09:30:24.471 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:24.639 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T09:30:24.655 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T09:30:24.655 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:24.655 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T09:30:24.655 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:26.142 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T09:30:26.168 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T09:30:26.168 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:26.168 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T09:30:26.168 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T09:30:26.168 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T09:30:26.168 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:26.394 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T09:30:26.417 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T09:30:26.418 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:26.418 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T09:30:26.418 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T09:30:26.418 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T09:30:26.418 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:26.870 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T09:30:26.892 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T09:30:26.893 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:26.893 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T09:30:26.893 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T09:30:26.893 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T09:30:26.893 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:26.981 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T09:30:27.398 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T09:30:27.446 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T09:30:27.478 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T09:30:27.538 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T09:30:27.561 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T09:30:27.795 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T09:30:27.820 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T09:30:28.787 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T09:30:28.812 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T09:30:28.911 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T09:30:29.076 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T09:30:29.090 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T09:30:29.090 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-03-16T09:30:29.090 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-03-16T09:30:29.090 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-03-16T09:30:29.090 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-03-16T09:30:29.090 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-03-16T09:30:29.090 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-03-16T09:30:29.091 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-03-16T09:30:29.091 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:29.106 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T09:30:29.130 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T09:30:29.130 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:29.131 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T09:30:29.131 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T09:30:29.131 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T09:30:29.131 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:29.174 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T09:30:29.185 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T09:30:29.247 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T09:30:29.260 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T09:30:29.299 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T09:30:29.299 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:29.299 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T09:30:29.299 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:29.609 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T09:30:29.609 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-03-16T09:30:29.609 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-03-16T09:30:29.609 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-03-16T09:30:29.610 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-03-16T09:30:29.610 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-03-16T09:30:29.610 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-03-16T09:30:29.610 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-03-16T09:30:29.610 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:29.776 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T09:30:29.792 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T09:30:29.792 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:29.792 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T09:30:29.792 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:30.121 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T09:30:30.151 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T09:30:30.174 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T09:30:30.174 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:30.174 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T09:30:30.175 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T09:30:30.175 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T09:30:30.175 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:30.219 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T09:30:30.240 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T09:30:30.240 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:30.240 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T09:30:30.240 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:30.286 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:30.308 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T09:30:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T09:30:30.308 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T09:30:30.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T09:30:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T09:30:30.643 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:30.643 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T09:30:30.643 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T09:30:30.643 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T09:30:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T09:30:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T09:30:30.780 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:30.780 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T09:30:30.780 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T09:30:30.780 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T09:30:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:31.005 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T09:30:31.028 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T09:30:31.028 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:31.028 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T09:30:31.028 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T09:30:31.028 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T09:30:31.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T09:30:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T09:30:31.280 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:31.281 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T09:30:31.281 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T09:30:31.281 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T09:30:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:31.464 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T09:30:31.487 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T09:30:31.487 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:31.487 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T09:30:31.487 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T09:30:31.487 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T09:30:31.487 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:31.505 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T09:30:31.527 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T09:30:31.528 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:31.528 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T09:30:31.528 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T09:30:31.528 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T09:30:31.528 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:31.574 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T09:30:31.967 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T09:30:31.972 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T09:30:31.995 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T09:30:31.995 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:31.995 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T09:30:31.995 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T09:30:31.995 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T09:30:31.995 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:31.997 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T09:30:32.029 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T09:30:32.082 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T09:30:32.089 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T09:30:32.112 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T09:30:32.344 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T09:30:32.388 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T09:30:32.457 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T09:30:32.489 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T09:30:32.521 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T09:30:32.580 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T09:30:32.604 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T09:30:32.837 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T09:30:32.872 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T09:30:33.350 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T09:30:33.380 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T09:30:33.480 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T09:30:33.643 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T09:30:33.674 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T09:30:33.697 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T09:30:33.697 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:33.697 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T09:30:33.697 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T09:30:33.698 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T09:30:33.698 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:33.741 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T09:30:33.752 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T09:30:33.800 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T09:30:33.834 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T09:30:33.855 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T09:30:33.954 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T09:30:34.118 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T09:30:34.149 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T09:30:34.172 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T09:30:34.172 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:34.172 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T09:30:34.172 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T09:30:34.172 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T09:30:34.172 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:34.216 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T09:30:34.227 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T09:30:34.277 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T09:30:34.663 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T09:30:34.693 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T09:30:34.716 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T09:30:34.716 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:34.716 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T09:30:34.716 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T09:30:34.716 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T09:30:34.716 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:34.760 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T09:30:34.782 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T09:30:34.783 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:34.783 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T09:30:34.783 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:34.827 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:34.850 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:34.850 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:34.850 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T09:30:34.850 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T09:30:34.850 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T09:30:34.850 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:35.091 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T09:30:35.138 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T09:30:35.147 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T09:30:35.153 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T09:30:35.172 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T09:30:35.175 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T09:30:35.175 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:35.176 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T09:30:35.176 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T09:30:35.176 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T09:30:35.176 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:35.176 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T09:30:35.199 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T09:30:35.199 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T09:30:35.199 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:35.199 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T09:30:35.199 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T09:30:35.199 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T09:30:35.200 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:35.244 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T09:30:35.266 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T09:30:35.266 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:35.266 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T09:30:35.266 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:35.311 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:35.333 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:35.333 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:35.334 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T09:30:35.334 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T09:30:35.334 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T09:30:35.334 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:35.382 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T09:30:35.500 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T09:30:35.539 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T09:30:35.595 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T09:30:35.653 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T09:30:35.664 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T09:30:35.675 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T09:30:35.675 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T09:30:35.675 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T09:30:35.675 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T09:30:35.676 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T09:30:35.676 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:35.699 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T09:30:35.732 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T09:30:35.755 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T09:30:35.785 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T09:30:35.854 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T09:30:35.930 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T09:30:35.967 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T09:30:36.529 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:36.566 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:36.586 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:37.394 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T09:30:37.470 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T09:30:37.511 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T09:30:37.605 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T09:30:37.658 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T09:30:37.683 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T09:30:37.708 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T09:30:37.734 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T09:30:37.735 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T09:30:37.822 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T09:30:37.861 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T09:30:37.861 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T09:30:37.945 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T09:30:37.971 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T09:30:38.005 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T09:30:38.031 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T09:30:38.055 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T09:30:38.099 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T09:30:38.145 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T09:30:38.177 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T09:30:38.202 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T09:30:38.227 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T09:30:38.253 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T09:30:38.278 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T09:30:38.285 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:38.285 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:38.465 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:38.465 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T09:30:38.547 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T09:30:38.572 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T09:30:38.597 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T09:30:38.630 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T09:30:38.657 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T09:30:38.722 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T09:30:39.616 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T09:30:39.663 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T09:30:39.696 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T09:30:39.723 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T09:30:39.904 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T09:30:40.023 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T09:30:40.063 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T09:30:40.125 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T09:30:40.149 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T09:30:40.196 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T09:30:40.199 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T09:30:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T09:30:40.239 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T09:30:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T09:30:40.272 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T09:30:40.304 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T09:30:40.334 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T09:30:40.400 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T09:30:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T09:30:40.465 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T09:30:40.507 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T09:30:40.572 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T09:30:40.615 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T09:30:40.671 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T09:30:40.723 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T09:30:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T09:30:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T09:30:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T09:30:41.038 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T09:30:41.038 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T09:30:41.038 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T09:30:41.052 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T09:30:41.095 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:41.131 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:41.150 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:41.625 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:41.665 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:41.685 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T09:30:41.978 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T09:30:42.058 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T09:30:42.090 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T09:30:42.176 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T09:30:42.211 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T09:30:42.236 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T09:30:42.262 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T09:30:42.288 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T09:30:42.288 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T09:30:42.376 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T09:30:42.405 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T09:30:42.406 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T09:30:42.489 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T09:30:42.490 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T09:30:42.525 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T09:30:42.549 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T09:30:42.568 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T09:30:42.575 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T09:30:42.600 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T09:30:42.609 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T09:30:42.645 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T09:30:42.689 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T09:30:42.690 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T09:30:42.722 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T09:30:42.729 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T09:30:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T09:30:42.754 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T09:30:42.772 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T09:30:42.780 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T09:30:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T09:30:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T09:30:42.805 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T09:30:42.831 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T09:30:42.837 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:42.837 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:42.886 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T09:30:42.916 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T09:30:42.916 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T09:30:43.001 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T09:30:43.017 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:43.017 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T09:30:43.027 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T09:30:43.060 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T09:30:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T09:30:43.091 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T09:30:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T09:30:43.116 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T09:30:43.141 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T09:30:43.153 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T09:30:43.174 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T09:30:43.192 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T09:30:43.201 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T09:30:43.223 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T09:30:43.248 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T09:30:43.266 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T09:30:43.274 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T09:30:43.299 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T09:30:43.324 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T09:30:43.331 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:43.331 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:43.511 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T09:30:43.511 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T09:30:43.593 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T09:30:43.616 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T09:30:43.617 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T09:30:43.618 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T09:30:43.619 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T09:30:43.620 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T09:30:43.621 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T09:30:43.622 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T09:30:43.623 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T09:30:43.624 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T09:30:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T09:30:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T09:30:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T09:30:43.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T09:30:43.629 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T09:30:43.630 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T09:30:43.631 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T09:30:43.631 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T09:30:43.631 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T09:30:43.643 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T09:30:43.676 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T09:30:43.703 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T09:30:43.768 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:46.857 INFO:teuthology.orchestra.run.smithi129.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.858 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T09:30:46.859 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.860 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.861 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T09:30:46.862 INFO:teuthology.orchestra.run.smithi129.stdout: jq-1.6-16.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T09:30:46.863 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T09:30:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T09:30:46.865 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T09:30:46.866 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T09:30:46.867 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T09:30:46.868 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T09:30:46.869 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.870 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T09:30:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:30:46.872 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-16T09:30:47.362 DEBUG:teuthology.parallel:result is None 2024-03-16T09:30:48.108 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T09:30:48.108 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T09:30:48.109 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T09:30:48.110 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T09:30:48.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T09:30:48.128 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T09:30:48.129 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T09:30:48.130 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T09:30:48.131 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T09:30:48.132 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T09:30:48.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T09:30:48.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:48.135 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T09:30:48.136 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T09:30:48.137 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T09:30:48.138 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T09:30:48.565 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T09:30:48.565 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T09:30:48.565 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T09:30:48.565 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T09:30:48.565 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T09:30:48.565 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T09:30:48.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T09:30:48.567 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T09:30:48.568 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T09:30:48.569 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T09:30:48.570 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T09:30:48.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T09:30:48.572 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T09:30:48.573 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T09:30:48.575 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T09:30:48.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T09:30:48.577 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T09:30:48.578 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T09:30:48.579 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T09:30:48.580 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T09:30:48.580 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T09:30:48.580 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T09:30:51.231 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.232 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.233 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.234 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-16.el9.x86_64 2024-03-16T09:30:51.235 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.236 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T09:30:51.237 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T09:30:51.238 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T09:30:51.239 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T09:30:51.240 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T09:30:51.241 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T09:30:51.242 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T09:30:51.243 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T09:30:51.244 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T09:30:51.244 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:30:51.244 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-03-16T09:30:51.539 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T09:30:51.539 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:51.539 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-03-16T09:30:51.539 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.539 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.539 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.539 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.540 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.541 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.542 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T09:30:51.543 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-16.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T09:30:51.544 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T09:30:51.545 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T09:30:51.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T09:30:51.547 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T09:30:51.548 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.549 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T09:30:51.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T09:30:51.551 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T09:30:51.552 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T09:30:51.552 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-16T09:30:51.552 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-16T09:30:51.753 DEBUG:teuthology.parallel:result is None 2024-03-16T09:30:52.038 DEBUG:teuthology.parallel:result is None 2024-03-16T09:30:52.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:30:52.183 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T09:30:52.206 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T09:30:52.206 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T09:30:52.206 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T09:30:52.208 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:30:52.348 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T09:30:52.369 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T09:30:52.370 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T09:30:52.370 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T09:30:52.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:30:52.514 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T09:30:52.540 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T09:30:52.540 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T09:30:52.540 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T09:30:52.542 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T09:30:52.543 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:30:52.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T09:30:52.578 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:30:52.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T09:30:52.609 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:30:52.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T09:30:52.644 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T09:30:52.644 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:30:52.644 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T09:30:52.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T09:30:52.742 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:30:52.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T09:30:52.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T09:30:52.837 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:30:52.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T09:30:52.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T09:30:52.938 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T09:30:52.939 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:30:52.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T09:30:52.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T09:30:53.036 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:30:53.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T09:30:53.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T09:30:53.131 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:30:53.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T09:30:53.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T09:30:53.231 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T09:30:53.231 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:30:53.231 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T09:30:53.260 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T09:30:53.329 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:30:53.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T09:30:53.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T09:30:53.423 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:30:53.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T09:30:53.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T09:30:53.521 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-16T09:30:53.622 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2}} 2024-03-16T09:30:53.622 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:30:53.624 INFO:tasks.cephadm:Cluster fsid is e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:30:53.624 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-16T09:30:53.624 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.77', 'mon.b': '172.21.15.119', 'mon.c': '172.21.15.129'} 2024-03-16T09:30:53.624 INFO:tasks.cephadm:First mon is mon.a on smithi077 2024-03-16T09:30:53.624 INFO:tasks.cephadm:First mgr is x 2024-03-16T09:30:53.624 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-16T09:30:53.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-03-16T09:30:53.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-03-16T09:30:53.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-03-16T09:30:53.712 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-16T09:30:53.712 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:30:53.852 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-16T09:30:53.997 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:30:53.999 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-16T09:30:53.999 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:54.432 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 09:30 /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:54.433 DEBUG:teuthology.orchestra.run.smithi119:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:54.918 INFO:teuthology.orchestra.run.smithi119.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 09:30 /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:54.918 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:55.512 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 09:30 /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:55.512 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:55.531 DEBUG:teuthology.orchestra.run.smithi119:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:55.550 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T09:30:55.585 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-16T09:30:55.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T09:30:55.587 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T09:30:55.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T09:30:56.515 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T09:30:56.538 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T09:30:56.557 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T09:31:30.198 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:31:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T09:31:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T09:31:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-03-16T09:31:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T09:31:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:31:30.199 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:31:42.308 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-16T09:31:42.308 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T09:31:42.308 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T09:31:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-03-16T09:31:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T09:31:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-03-16T09:31:42.309 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-16T09:31:43.582 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-16T09:31:43.582 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T09:31:43.582 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T09:31:43.583 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-03-16T09:31:43.583 INFO:teuthology.orchestra.run.smithi119.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T09:31:43.583 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-16T09:31:43.583 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-16T09:31:43.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-03-16T09:31:43.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-03-16T09:31:43.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-03-16T09:31:43.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-03-16T09:31:43.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-03-16T09:31:43.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-03-16T09:31:43.798 INFO:tasks.cephadm:Writing seed config... 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-16T09:31:43.799 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-16T09:31:43.800 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-16T09:31:43.801 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-16T09:31:43.802 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-16T09:31:43.802 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:31:43.802 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-16T09:31:43.819 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e287e1d6-e377-11ee-95c9-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-16T09:31:43.820 DEBUG:teuthology.orchestra.run.smithi077:mon.a> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a.service 2024-03-16T09:31:43.862 DEBUG:teuthology.orchestra.run.smithi077:mgr.x> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x.service 2024-03-16T09:31:43.905 INFO:tasks.cephadm:Bootstrapping... 2024-03-16T09:31:43.906 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid e287e1d6-e377-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.77 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:31:44.103 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:31:44.103 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', 'e287e1d6-e377-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.77', '--skip-admin-label'] 2024-03-16T09:31:44.128 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T09:31:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-16T09:31:44.129 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying podman|docker is present... 2024-03-16T09:31:44.151 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T09:31:44.151 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying lvm2 is present... 2024-03-16T09:31:44.151 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying time synchronization is in place... 2024-03-16T09:31:44.159 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T09:31:44.159 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T09:31:44.165 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T09:31:44.165 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T09:31:44.172 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout enabled 2024-03-16T09:31:44.178 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout active 2024-03-16T09:31:44.178 INFO:teuthology.orchestra.run.smithi077.stdout:Unit chronyd.service is enabled and running 2024-03-16T09:31:44.178 INFO:teuthology.orchestra.run.smithi077.stdout:Repeating the final host check... 2024-03-16T09:31:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T09:31:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-16T09:31:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl is present 2024-03-16T09:31:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:lvcreate is present 2024-03-16T09:31:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T09:31:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T09:31:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T09:31:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T09:31:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout enabled 2024-03-16T09:31:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout active 2024-03-16T09:31:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:Unit chronyd.service is enabled and running 2024-03-16T09:31:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:Host looks OK 2024-03-16T09:31:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:Cluster fsid: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:31:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:Acquiring lock 140337995927456 on /run/cephadm/e287e1d6-e377-11ee-95c9-87774f69a715.lock 2024-03-16T09:31:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:Lock 140337995927456 acquired on /run/cephadm/e287e1d6-e377-11ee-95c9-87774f69a715.lock 2024-03-16T09:31:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying IP 172.21.15.77 port 3300 ... 2024-03-16T09:31:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying IP 172.21.15.77 port 6789 ... 2024-03-16T09:31:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:Base mon IP(s) is [172.21.15.77:3300, 172.21.15.77:6789], mon addrv is [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-03-16T09:31:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.77 metric 100 2024-03-16T09:31:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.77 metric 100 2024-03-16T09:31:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-16T09:31:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-16T09:31:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1733sec hoplimit 64 pref medium 2024-03-16T09:31:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-16T09:31:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-16T09:31:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T09:31:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-16T09:31:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-16T09:31:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T09:31:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-03-16T09:31:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-03-16T09:31:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-16T09:31:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-16T09:31:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T09:31:44.744 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T09:31:44.744 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T09:31:44.744 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Getting image source signatures 2024-03-16T09:31:44.745 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-16T09:31:44.745 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-16T09:31:44.745 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T09:31:44.745 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-16T09:31:45.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T09:31:45.675 INFO:teuthology.orchestra.run.smithi077.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T09:31:45.675 INFO:teuthology.orchestra.run.smithi077.stdout:Extracting ceph user uid/gid from container image... 2024-03-16T09:31:49.979 INFO:teuthology.orchestra.run.smithi077.stdout:stat: stdout 167 167 2024-03-16T09:31:49.980 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial keys... 2024-03-16T09:31:55.883 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQAIZ/VldsO5LhAAgszBXHD7neZ8D5kz5zKcvw== 2024-03-16T09:31:56.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQAMZ/VlYP5PDBAAGrLg2MO1JpM/bTMu5PByLw== 2024-03-16T09:31:57.668 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQANZ/Vlkw65BRAAtKMQAuEMoo9cQ7ncJfHOXg== 2024-03-16T09:31:57.669 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial monmap... 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool for a [v2:172.21.15.77:3300,v1:172.21.15.77:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = quincy 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: set fsid to e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:31:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T09:31:58.643 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:31:58.643 INFO:teuthology.orchestra.run.smithi077.stdout:Creating mon... 2024-03-16T09:31:59.646 INFO:teuthology.orchestra.run.smithi077.stdout:create mon.a on 2024-03-16T09:31:59.855 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-16T09:31:59.989 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-16T09:32:00.152 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e287e1d6-e377-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-e287e1d6-e377-11ee-95c9-87774f69a715.target. 2024-03-16T09:32:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e287e1d6-e377-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-e287e1d6-e377-11ee-95c9-87774f69a715.target. 2024-03-16T09:32:00.388 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a 2024-03-16T09:32:00.388 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-16T09:32:00.551 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e287e1d6-e377-11ee-95c9-87774f69a715.target.wants/ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e287e1d6-e377-11ee-95c9-87774f69a715@.service. 2024-03-16T09:32:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:00 smithi077 podman[31766]: 2024-03-16 09:32:00.696681907 +0000 UTC m=+0.017686813 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:32:02.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:01 smithi077 podman[31766]: 2024-03-16 09:32:01.874681582 +0000 UTC m=+1.195686480 container create de4f3fe0601b53b65a39229a032de6b98f67c391bb3c08d03b0849a56b16b843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T09:32:02.899 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T09:32:02.899 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T09:32:02.906 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T09:32:02.906 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T09:32:02.906 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T09:32:02.906 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T09:32:02.907 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mon to start... 2024-03-16T09:32:02.907 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mon... 2024-03-16T09:32:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:02 smithi077 podman[31766]: 2024-03-16 09:32:02.808473118 +0000 UTC m=+2.129478016 container init de4f3fe0601b53b65a39229a032de6b98f67c391bb3c08d03b0849a56b16b843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T09:32:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:02 smithi077 podman[31766]: 2024-03-16 09:32:02.811889121 +0000 UTC m=+2.132894016 container start de4f3fe0601b53b65a39229a032de6b98f67c391bb3c08d03b0849a56b16b843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-16T09:32:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:02 smithi077 ceph-mon[31797]: mkfs e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:02 smithi077 bash[31766]: de4f3fe0601b53b65a39229a032de6b98f67c391bb3c08d03b0849a56b16b843 2024-03-16T09:32:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:02 smithi077 ceph-mon[31797]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T09:32:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:02 smithi077 systemd[1]: Started Ceph mon.a for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:32:03.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T09:32:03.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: monmap epoch 1 2024-03-16T09:32:03.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:03.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: last_changed 2024-03-16T09:31:58.056995+0000 2024-03-16T09:32:03.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: created 2024-03-16T09:31:58.056995+0000 2024-03-16T09:32:03.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: min_mon_release 19 (squid) 2024-03-16T09:32:03.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: election_strategy: 1 2024-03-16T09:32:03.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T09:32:03.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: fsmap 2024-03-16T09:32:03.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T09:32:03.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: mgrmap e1: no daemons active 2024-03-16T09:32:03.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:03 smithi077 ceph-mon[31797]: from='client.? 172.21.15.77:0/4048453576' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-16T09:32:03.977 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout cluster: 2024-03-16T09:32:03.983 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout id: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout services: 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.523073s) 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout data: 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-16T09:32:03.984 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout pgs: 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.324+0000 7f7b0a914700 1 Processor -- start 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.324+0000 7f7b0a914700 1 -- start start 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.324+0000 7f7b0a914700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b04108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.324+0000 7f7b0a914700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041086f0 con 0x7f7b04107d00 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b04108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b04108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57254/0 (socket says 172.21.15.77:57254) 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b03fff700 1 -- 172.21.15.77:0/1161939846 learned_addr learned my addr 172.21.15.77:0/1161939846 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b03fff700 1 -- 172.21.15.77:0/1161939846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b04108f00 con 0x7f7b04107d00 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b03fff700 1 --2- 172.21.15.77:0/1161939846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b04108120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7af40098a0 tx=0x7f7af4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cdcd4b078b63c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b02ffd700 1 -- 172.21.15.77:0/1161939846 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af4004160 con 0x7f7b04107d00 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b02ffd700 1 -- 172.21.15.77:0/1161939846 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af4004300 con 0x7f7b04107d00 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.325+0000 7f7b02ffd700 1 -- 172.21.15.77:0/1161939846 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af40036a0 con 0x7f7b04107d00 2024-03-16T09:32:03.985 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 -- 172.21.15.77:0/1161939846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 msgr2=0x7f7b04108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 --2- 172.21.15.77:0/1161939846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b04108120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7af40098a0 tx=0x7f7af4009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 -- 172.21.15.77:0/1161939846 shutdown_connections 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 --2- 172.21.15.77:0/1161939846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b04108120 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 -- 172.21.15.77:0/1161939846 >> 172.21.15.77:0/1161939846 conn(0x7f7b04103340 msgr2=0x7f7b04105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 -- 172.21.15.77:0/1161939846 shutdown_connections 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 -- 172.21.15.77:0/1161939846 wait complete. 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 Processor -- start 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.326+0000 7f7b0a914700 1 -- start start 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b0a914700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b0419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b0a914700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04109430 con 0x7f7b04107d00 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b0419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b0419b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57262/0 (socket says 172.21.15.77:57262) 2024-03-16T09:32:03.986 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b03fff700 1 -- 172.21.15.77:0/4048453576 learned_addr learned my addr 172.21.15.77:0/4048453576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b03fff700 1 -- 172.21.15.77:0/4048453576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0419b810 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b03fff700 1 --2- 172.21.15.77:0/4048453576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b0419b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7af4000c00 tx=0x7f7af4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b017fa700 1 -- 172.21.15.77:0/4048453576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af4003db0 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.327+0000 7f7b017fa700 1 -- 172.21.15.77:0/4048453576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af4011630 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.328+0000 7f7b017fa700 1 -- 172.21.15.77:0/4048453576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af4011950 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.328+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0419baa0 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.328+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0419e790 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.328+0000 7f7b017fa700 1 -- 172.21.15.77:0/4048453576 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7af4011af0 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.328+0000 7f7b017fa700 1 -- 172.21.15.77:0/4048453576 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7af401c450 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.328+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b04063900 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.331+0000 7f7b017fa700 1 -- 172.21.15.77:0/4048453576 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7af4032020 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.400+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f7b0402dcf0 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.400+0000 7f7b017fa700 1 -- 172.21.15.77:0/4048453576 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f7af4004260 con 0x7f7b04107d00 2024-03-16T09:32:03.987 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.401+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 msgr2=0x7f7b0419b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.401+0000 7f7b0a914700 1 --2- 172.21.15.77:0/4048453576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b0419b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7af4000c00 tx=0x7f7af4004750 comp rx=0 tx=0).stop 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.401+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 shutdown_connections 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.401+0000 7f7b0a914700 1 --2- 172.21.15.77:0/4048453576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b04107d00 0x7f7b0419b2d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.401+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 >> 172.21.15.77:0/4048453576 conn(0x7f7b04103340 msgr2=0x7f7b04191f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.401+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 shutdown_connections 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:03.401+0000 7f7b0a914700 1 -- 172.21.15.77:0/4048453576 wait complete. 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:mon is available 2024-03-16T09:32:03.988 INFO:teuthology.orchestra.run.smithi077.stdout:Assimilating anything we can from ceph.conf... 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout fsid = e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T09:32:04.998 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.413+0000 7f1d206c2700 1 Processor -- start 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.413+0000 7f1d206c2700 1 -- start start 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.413+0000 7f1d206c2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:04.999 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.413+0000 7f1d206c2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d181075b0 con 0x7f1d18106bc0 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1e45e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1e45e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57266/0 (socket says 172.21.15.77:57266) 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1e45e700 1 -- 172.21.15.77:0/226325305 learned_addr learned my addr 172.21.15.77:0/226325305 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1e45e700 1 -- 172.21.15.77:0/226325305 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d18107dc0 con 0x7f1d18106bc0 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1e45e700 1 --2- 172.21.15.77:0/226325305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18106fe0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000bb40 tx=0x7f1d1000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb91859f2f740c32 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1d45c700 1 -- 172.21.15.77:0/226325305 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d1000dbe0 con 0x7f1d18106bc0 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1d45c700 1 -- 172.21.15.77:0/226325305 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1000dd80 con 0x7f1d18106bc0 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.414+0000 7f1d1d45c700 1 -- 172.21.15.77:0/226325305 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d10004180 con 0x7f1d18106bc0 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 -- 172.21.15.77:0/226325305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 msgr2=0x7f1d18106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 --2- 172.21.15.77:0/226325305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18106fe0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000bb40 tx=0x7f1d1000be50 comp rx=0 tx=0).stop 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 -- 172.21.15.77:0/226325305 shutdown_connections 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 --2- 172.21.15.77:0/226325305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18106fe0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 -- 172.21.15.77:0/226325305 >> 172.21.15.77:0/226325305 conn(0x7f1d18102180 msgr2=0x7f1d181045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:05.000 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 -- 172.21.15.77:0/226325305 shutdown_connections 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 -- 172.21.15.77:0/226325305 wait complete. 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.415+0000 7f1d206c2700 1 Processor -- start 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d206c2700 1 -- start start 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d206c2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d206c2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d18108230 con 0x7f1d18106bc0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d1e45e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d1e45e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57268/0 (socket says 172.21.15.77:57268) 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d1e45e700 1 -- 172.21.15.77:0/3637312504 learned_addr learned my addr 172.21.15.77:0/3637312504 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d1e45e700 1 -- 172.21.15.77:0/3637312504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d181995d0 con 0x7f1d18106bc0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d1e45e700 1 --2- 172.21.15.77:0/3637312504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18199090 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000b340 tx=0x7f1d10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d10029070 con 0x7f1d18106bc0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d100044e0 con 0x7f1d18106bc0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d18199860 con 0x7f1d18106bc0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d100105a0 con 0x7f1d18106bc0 2024-03-16T09:32:05.001 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.416+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1819c550 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.417+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1d1002c070 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.417+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1d10021070 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.417+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d18063900 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.419+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1d10015070 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.486+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1d1802dcc0 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.489+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+456 (secure 0 0 0) 0x7f1d1000dce0 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.489+0000 7f1d0f7fe700 1 -- 172.21.15.77:0/3637312504 <== mon.0 v2:172.21.15.77:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1d10010bd0 con 0x7f1d18106bc0 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.490+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 msgr2=0x7f1d18199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.490+0000 7f1d206c2700 1 --2- 172.21.15.77:0/3637312504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18199090 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000b340 tx=0x7f1d10004750 comp rx=0 tx=0).stop 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.490+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 shutdown_connections 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.490+0000 7f1d206c2700 1 --2- 172.21.15.77:0/3637312504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d18106bc0 0x7f1d18199090 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.490+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 >> 172.21.15.77:0/3637312504 conn(0x7f1d18102180 msgr2=0x7f1d1818fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:05.002 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.490+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 shutdown_connections 2024-03-16T09:32:05.003 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:04.490+0000 7f1d206c2700 1 -- 172.21.15.77:0/3637312504 wait complete. 2024-03-16T09:32:05.003 INFO:teuthology.orchestra.run.smithi077.stdout:Generating new minimal ceph.conf... 2024-03-16T09:32:05.098 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:04 smithi077 ceph-mon[31797]: from='client.? 172.21.15.77:0/3637312504' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T09:32:05.098 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:04 smithi077 ceph-mon[31797]: from='client.? 172.21.15.77:0/3637312504' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T09:32:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:05 smithi077 ceph-mon[31797]: from='client.? 172.21.15.77:0/352720618' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:32:06.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.368+0000 7f1af0896700 1 Processor -- start 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.368+0000 7f1af0896700 1 -- start start 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.368+0000 7f1af0896700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec109740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.368+0000 7f1af0896700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec109d10 con 0x7f1aec109320 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.368+0000 7f1aea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.368+0000 7f1aea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec109740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57270/0 (socket says 172.21.15.77:57270) 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.368+0000 7f1aea59c700 1 -- 172.21.15.77:0/2942386259 learned_addr learned my addr 172.21.15.77:0/2942386259 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.369+0000 7f1aea59c700 1 -- 172.21.15.77:0/2942386259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec109e90 con 0x7f1aec109320 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.369+0000 7f1aea59c700 1 --2- 172.21.15.77:0/2942386259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec109740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0098a0 tx=0x7f1adc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffafbddc56f88fa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.369+0000 7f1ae959a700 1 -- 172.21.15.77:0/2942386259 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1adc004160 con 0x7f1aec109320 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.369+0000 7f1ae959a700 1 -- 172.21.15.77:0/2942386259 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1adc004300 con 0x7f1aec109320 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.369+0000 7f1ae959a700 1 -- 172.21.15.77:0/2942386259 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1adc011a80 con 0x7f1aec109320 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 -- 172.21.15.77:0/2942386259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 msgr2=0x7f1aec109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:06.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 --2- 172.21.15.77:0/2942386259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec109740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0098a0 tx=0x7f1adc009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 -- 172.21.15.77:0/2942386259 shutdown_connections 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 --2- 172.21.15.77:0/2942386259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec109740 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 -- 172.21.15.77:0/2942386259 >> 172.21.15.77:0/2942386259 conn(0x7f1aec101980 msgr2=0x7f1aec103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 -- 172.21.15.77:0/2942386259 shutdown_connections 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 -- 172.21.15.77:0/2942386259 wait complete. 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 Processor -- start 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.370+0000 7f1af0896700 1 -- start start 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1af0896700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1af0896700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec106ac0 con 0x7f1aec109320 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1aea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:06.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1aea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57282/0 (socket says 172.21.15.77:57282) 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1aea59c700 1 -- 172.21.15.77:0/352720618 learned_addr learned my addr 172.21.15.77:0/352720618 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1aea59c700 1 -- 172.21.15.77:0/352720618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec19c000 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1aea59c700 1 --2- 172.21.15.77:0/352720618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec19bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1adc000c00 tx=0x7f1adc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1adb7fe700 1 -- 172.21.15.77:0/352720618 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1adc0039f0 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1adb7fe700 1 -- 172.21.15.77:0/352720618 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1adc003b90 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aec19c2f0 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1adb7fe700 1 -- 172.21.15.77:0/352720618 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1adc01a740 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aec19ef80 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1adb7fe700 1 -- 172.21.15.77:0/352720618 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1adc02d3f0 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.371+0000 7f1adb7fe700 1 -- 172.21.15.77:0/352720618 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1adc019a40 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.372+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aec063900 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.374+0000 7f1adb7fe700 1 -- 172.21.15.77:0/352720618 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1adc024af0 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.443+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1aec19f5a0 con 0x7f1aec109320 2024-03-16T09:32:06.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.443+0000 7f1adb7fe700 1 -- 172.21.15.77:0/352720618 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f1adc019c40 con 0x7f1aec109320 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.444+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 msgr2=0x7f1aec19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.444+0000 7f1af0896700 1 --2- 172.21.15.77:0/352720618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec19bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1adc000c00 tx=0x7f1adc011b00 comp rx=0 tx=0).stop 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.444+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 shutdown_connections 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.444+0000 7f1af0896700 1 --2- 172.21.15.77:0/352720618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec109320 0x7f1aec19bac0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.444+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 >> 172.21.15.77:0/352720618 conn(0x7f1aec101980 msgr2=0x7f1aec102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.444+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 shutdown_connections 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:05.444+0000 7f1af0896700 1 -- 172.21.15.77:0/352720618 wait complete. 2024-03-16T09:32:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:Restarting the monitor... 2024-03-16T09:32:06.308 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:06 smithi077 systemd[1]: Stopping Ceph mon.a for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:32:06.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:06 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a[31792]: 2024-03-16T09:32:06.305+0000 7fc49e134700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:32:06.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:06 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a[31792]: 2024-03-16T09:32:06.305+0000 7fc49e134700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T09:32:06.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:06 smithi077 podman[32044]: 2024-03-16 09:32:06.438696729 +0000 UTC m=+0.242533475 container died de4f3fe0601b53b65a39229a032de6b98f67c391bb3c08d03b0849a56b16b843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T09:32:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:06 smithi077 podman[32044]: 2024-03-16 09:32:06.813440976 +0000 UTC m=+0.617277720 container cleanup de4f3fe0601b53b65a39229a032de6b98f67c391bb3c08d03b0849a56b16b843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T09:32:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:06 smithi077 bash[32044]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a 2024-03-16T09:32:07.369 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 podman[32058]: 2024-03-16 09:32:07.030951438 +0000 UTC m=+0.593682848 container remove de4f3fe0601b53b65a39229a032de6b98f67c391bb3c08d03b0849a56b16b843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:32:07.369 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 systemd[1]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-16T09:32:07.369 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 systemd[1]: Stopped Ceph mon.a for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:32:07.369 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 systemd[1]: Starting Ceph mon.a for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:32:07.662 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 podman[32152]: 2024-03-16 09:32:07.269554902 +0000 UTC m=+0.017079090 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:32:07.662 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 podman[32152]: 2024-03-16 09:32:07.373516187 +0000 UTC m=+0.121040416 container create 80a2c43e44ea6f70a7a1b9f0990c41d0171778fa970bbbc1f6d3fd94ee4a5946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-16T09:32:07.662 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 podman[32152]: 2024-03-16 09:32:07.576611391 +0000 UTC m=+0.324135591 container init 80a2c43e44ea6f70a7a1b9f0990c41d0171778fa970bbbc1f6d3fd94ee4a5946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:32:07.662 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 podman[32152]: 2024-03-16 09:32:07.579670978 +0000 UTC m=+0.327195171 container start 80a2c43e44ea6f70a7a1b9f0990c41d0171778fa970bbbc1f6d3fd94ee4a5946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:32:07.663 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T09:32:07.663 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T09:32:07.663 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: pidfile_write: ignore empty --pid-file 2024-03-16T09:32:07.663 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: load: jerasure load: lrc 2024-03-16T09:32:07.663 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: RocksDB version: 7.9.2 2024-03-16T09:32:07.664 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Git sha 0 2024-03-16T09:32:07.664 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T09:32:07.664 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: DB SUMMARY 2024-03-16T09:32:07.664 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: DB Session ID: YXQCFMYWKPCPGMOLMHBT 2024-03-16T09:32:07.664 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: CURRENT file: CURRENT 2024-03-16T09:32:07.664 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T09:32:07.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-16T09:32:07.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-16T09:32:07.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104110 ; 2024-03-16T09:32:07.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.error_if_exists: 0 2024-03-16T09:32:07.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.create_if_missing: 0 2024-03-16T09:32:07.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.paranoid_checks: 1 2024-03-16T09:32:07.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.env: 0x556609aeec60 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.info_log: 0x55660b6a3ac0 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.statistics: (nil) 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.use_fsync: 0 2024-03-16T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_log_file_size: 0 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.allow_fallocate: 1 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T09:32:07.667 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.use_direct_reads: 0 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.db_log_dir: 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.wal_dir: 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T09:32:07.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T09:32:07.669 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T09:32:07.669 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T09:32:07.669 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T09:32:07.669 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.write_buffer_manager: 0x55660b6761e0 2024-03-16T09:32:07.669 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T09:32:07.669 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T09:32:07.669 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T09:32:07.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T09:32:07.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T09:32:07.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T09:32:07.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T09:32:07.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T09:32:07.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.unordered_write: 0 2024-03-16T09:32:07.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.row_cache: None 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.wal_filter: None 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.two_write_queues: 0 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.wal_compression: 0 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.atomic_flush: 0 2024-03-16T09:32:07.671 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.log_readahead_size: 0 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_background_jobs: 2 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_background_compactions: -1 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_subcompactions: 1 2024-03-16T09:32:07.672 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_open_files: -1 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_background_flushes: -1 2024-03-16T09:32:07.673 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Compression algorithms supported: 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kZSTD supported: 0 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kXpressCompression supported: 0 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kZlibCompression supported: 1 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kSnappyCompression supported: 1 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kLZ4Compression supported: 1 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: kBZip2Compression supported: 0 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T09:32:07.674 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.merge_operator: 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_filter: None 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55660b6a3bc0) 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_type: 0 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_index_type: 0 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_shortening: 1 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T09:32:07.675 INFO:journalctl@ceph.mon.a.smithi077.stdout: checksum: 4 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: no_block_cache: 0 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache: 0x55660b6a9090 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_options: 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: capacity : 536870912 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: num_shard_bits : 4 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: strict_capacity_limit : 0 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_compressed: (nil) 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: persistent_cache: (nil) 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size: 4096 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size_deviation: 10 2024-03-16T09:32:07.676 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_restart_interval: 16 2024-03-16T09:32:07.677 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_block_restart_interval: 1 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: metadata_block_size: 4096 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: partition_filters: 0 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_delta_encoding: 1 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: filter_policy: bloomfilter 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: whole_key_filtering: 1 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: verify_compression: 0 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: format_version: 5 2024-03-16T09:32:07.678 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_index_compression: 1 2024-03-16T09:32:07.679 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_align: 0 2024-03-16T09:32:07.679 INFO:journalctl@ceph.mon.a.smithi077.stdout: max_auto_readahead_size: 262144 2024-03-16T09:32:07.679 INFO:journalctl@ceph.mon.a.smithi077.stdout: prepopulate_block_cache: 0 2024-03-16T09:32:07.679 INFO:journalctl@ceph.mon.a.smithi077.stdout: initial_auto_readahead_size: 8192 2024-03-16T09:32:07.679 INFO:journalctl@ceph.mon.a.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T09:32:07.679 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T09:32:07.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T09:32:07.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression: NoCompression 2024-03-16T09:32:07.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T09:32:07.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T09:32:07.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T09:32:07.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.num_levels: 7 2024-03-16T09:32:07.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T09:32:07.681 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T09:32:07.681 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T09:32:07.681 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T09:32:07.681 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T09:32:07.681 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T09:32:07.681 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T09:32:07.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T09:32:07.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T09:32:07.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T09:32:07.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T09:32:07.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T09:32:07.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T09:32:07.683 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T09:32:07.683 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T09:32:07.683 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T09:32:07.683 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T09:32:07.683 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T09:32:07.683 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T09:32:07.684 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T09:32:07.684 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T09:32:07.684 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T09:32:07.684 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T09:32:07.684 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T09:32:07.684 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T09:32:07.684 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T09:32:07.685 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T09:32:07.685 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T09:32:07.685 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T09:32:07.685 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T09:32:07.685 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T09:32:07.685 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T09:32:07.686 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T09:32:07.686 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T09:32:07.686 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T09:32:07.686 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T09:32:07.686 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T09:32:07.686 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T09:32:07.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T09:32:07.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T09:32:07.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T09:32:07.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T09:32:07.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T09:32:07.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T09:32:07.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T09:32:07.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T09:32:07.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T09:32:07.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T09:32:07.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T09:32:07.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T09:32:07.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T09:32:07.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T09:32:07.689 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T09:32:07.689 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.table_properties_collectors: 2024-03-16T09:32:07.689 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.inplace_update_support: 0 2024-03-16T09:32:07.689 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T09:32:07.689 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T09:32:07.689 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.bloom_locality: 0 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.max_successive_merges: 0 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.ttl: 2592000 2024-03-16T09:32:07.690 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.enable_blob_files: false 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.min_blob_size: 0 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T09:32:07.691 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 64e8ed15-2747-4f65-982c-c01ffb946381 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710581527608857, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-16T09:32:07.692 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710581527609953, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99299, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97284, "index_size": 297, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13935, "raw_average_key_size": 49, "raw_value_size": 89660, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710581527, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "64e8ed15-2747-4f65-982c-c01ffb946381", "db_session_id": "YXQCFMYWKPCPGMOLMHBT", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710581527610036, "job": 1, "event": "recovery_finished"} 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55660b79c000 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: DB pointer 0x55660b786000 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T09:32:07.693 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** DB Stats ** 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T09:32:07.694 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: L0 2/0 98.81 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: Sum 2/0 98.81 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T09:32:07.695 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T09:32:07.696 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative compaction: 0.00 GB write, 31.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval compaction: 0.00 GB write, 31.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: Block cache BinnedLRUCache@0x55660b6a9090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-16T09:32:07.697 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: starting mon.a rank 0 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???) e1 preinit fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???).mds e1 new map 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???).mds e1 print_map 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout: e1 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T09:32:07.698 INFO:journalctl@ceph.mon.a.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout: legacy client fscid: -1 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout: No filesystems configured 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-16T09:32:07.699 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 bash[32152]: 80a2c43e44ea6f70a7a1b9f0990c41d0171778fa970bbbc1f6d3fd94ee4a5946 2024-03-16T09:32:07.700 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 systemd[1]: Started Ceph mon.a for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:32:07.700 INFO:teuthology.orchestra.run.smithi077.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-16T09:32:08.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T09:32:08.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: monmap epoch 1 2024-03-16T09:32:08.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:08.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: last_changed 2024-03-16T09:31:58.056995+0000 2024-03-16T09:32:08.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: created 2024-03-16T09:31:58.056995+0000 2024-03-16T09:32:08.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: min_mon_release 19 (squid) 2024-03-16T09:32:08.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: election_strategy: 1 2024-03-16T09:32:08.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T09:32:08.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: fsmap 2024-03-16T09:32:08.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T09:32:08.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:07 smithi077 ceph-mon[32171]: mgrmap e1: no daemons active 2024-03-16T09:32:09.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.566+0000 7fb970d50700 1 Processor -- start 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.566+0000 7fb970d50700 1 -- start start 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.566+0000 7fb970d50700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c1069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.566+0000 7fb970d50700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c07bdb0 con 0x7fb96c1065c0 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.566+0000 7fb96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c1069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.566+0000 7fb96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c1069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40422/0 (socket says 172.21.15.77:40422) 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.566+0000 7fb96a59c700 1 -- 172.21.15.77:0/3297234641 learned_addr learned my addr 172.21.15.77:0/3297234641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.567+0000 7fb96a59c700 1 -- 172.21.15.77:0/3297234641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c106f20 con 0x7fb96c1065c0 2024-03-16T09:32:09.233 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.567+0000 7fb96a59c700 1 --2- 172.21.15.77:0/3297234641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c1069e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb9540098a0 tx=0x7fb954009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b4f95baded402f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.567+0000 7fb96959a700 1 -- 172.21.15.77:0/3297234641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb954004160 con 0x7fb96c1065c0 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.567+0000 7fb96959a700 1 -- 172.21.15.77:0/3297234641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb954004300 con 0x7fb96c1065c0 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.567+0000 7fb96959a700 1 -- 172.21.15.77:0/3297234641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb954011a80 con 0x7fb96c1065c0 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 -- 172.21.15.77:0/3297234641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 msgr2=0x7fb96c1069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 --2- 172.21.15.77:0/3297234641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c1069e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb9540098a0 tx=0x7fb954009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 -- 172.21.15.77:0/3297234641 shutdown_connections 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 --2- 172.21.15.77:0/3297234641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c1069e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 -- 172.21.15.77:0/3297234641 >> 172.21.15.77:0/3297234641 conn(0x7fb96c102200 msgr2=0x7fb96c104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 -- 172.21.15.77:0/3297234641 shutdown_connections 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 -- 172.21.15.77:0/3297234641 wait complete. 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.568+0000 7fb970d50700 1 Processor -- start 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb970d50700 1 -- start start 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb970d50700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb970d50700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c107e70 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c19dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40432/0 (socket says 172.21.15.77:40432) 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb96a59c700 1 -- 172.21.15.77:0/2096357357 learned_addr learned my addr 172.21.15.77:0/2096357357 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb96a59c700 1 -- 172.21.15.77:0/2096357357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c19e240 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb96a59c700 1 --2- 172.21.15.77:0/2096357357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c19dd00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb954006af0 tx=0x7fb954011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb9637fe700 1 -- 172.21.15.77:0/2096357357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9540039e0 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb9637fe700 1 -- 172.21.15.77:0/2096357357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb954003b80 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96c19e4d0 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb9637fe700 1 -- 172.21.15.77:0/2096357357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9540196a0 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.569+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96c1a11b0 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.570+0000 7fb9637fe700 1 -- 172.21.15.77:0/2096357357 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb95402d440 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.570+0000 7fb9637fe700 1 -- 172.21.15.77:0/2096357357 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb9540247c0 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.570+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb96c063900 con 0x7fb96c1065c0 2024-03-16T09:32:09.235 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.573+0000 7fb9637fe700 1 -- 172.21.15.77:0/2096357357 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb954043dd0 con 0x7fb96c1065c0 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.639+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fb96c02de30 con 0x7fb96c1065c0 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.641+0000 7fb9637fe700 1 -- 172.21.15.77:0/2096357357 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fb954032020 con 0x7fb96c1065c0 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.642+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 msgr2=0x7fb96c19dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.642+0000 7fb970d50700 1 --2- 172.21.15.77:0/2096357357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c19dd00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb954006af0 tx=0x7fb954011c40 comp rx=0 tx=0).stop 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.642+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 shutdown_connections 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.642+0000 7fb970d50700 1 --2- 172.21.15.77:0/2096357357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb96c1065c0 0x7fb96c19dd00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.642+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 >> 172.21.15.77:0/2096357357 conn(0x7fb96c102200 msgr2=0x7fb96c1949e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.642+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 shutdown_connections 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:08.642+0000 7fb970d50700 1 -- 172.21.15.77:0/2096357357 wait complete. 2024-03-16T09:32:09.236 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-16T09:32:09.251 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:32:09.251 INFO:teuthology.orchestra.run.smithi077.stdout:Creating mgr... 2024-03-16T09:32:09.251 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-16T09:32:09.251 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-16T09:32:09.468 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x 2024-03-16T09:32:09.468 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-16T09:32:09.625 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e287e1d6-e377-11ee-95c9-87774f69a715.target.wants/ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-e287e1d6-e377-11ee-95c9-87774f69a715@.service. 2024-03-16T09:32:09.966 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:09 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2096357357' entity='client.admin' 2024-03-16T09:32:10.192 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T09:32:10.192 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T09:32:10.199 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T09:32:10.200 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T09:32:10.200 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T09:32:10.200 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T09:32:10.209 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T09:32:10.209 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T09:32:10.217 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T09:32:10.217 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T09:32:10.217 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T09:32:10.217 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-16T09:32:10.217 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr to start... 2024-03-16T09:32:10.217 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr... 2024-03-16T09:32:10.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:10 smithi077 systemd[1]: Started Ceph mgr.x for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:32:11.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:10 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3103351222' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T09:32:11.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:10 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:10.633+0000 7f9e08d42200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T09:32:11.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:10 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:10.731+0000 7f9e08d42200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T09:32:11.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:11.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:11.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:32:11.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T09:32:11.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T09:32:11.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T09:32:11.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T09:32:11.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T09:32:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T09:32:11.362 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T09:32:11.363 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T09:32:02.879942+0000", 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.646+0000 7f1c2cd41700 1 Processor -- start 2024-03-16T09:32:11.364 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.646+0000 7f1c2cd41700 1 -- start start 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.646+0000 7f1c2cd41700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2810ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.646+0000 7f1c2cd41700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2807bdf0 con 0x7f1c28108750 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.647+0000 7f1c2659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2810ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.647+0000 7f1c2659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2810ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40458/0 (socket says 172.21.15.77:40458) 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.647+0000 7f1c2659c700 1 -- 172.21.15.77:0/414869190 learned_addr learned my addr 172.21.15.77:0/414869190 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.647+0000 7f1c2659c700 1 -- 172.21.15.77:0/414869190 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2810b080 con 0x7f1c28108750 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.647+0000 7f1c2659c700 1 --2- 172.21.15.77:0/414869190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2810ab40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0098a0 tx=0x7f1c1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb146b1486b6451b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.647+0000 7f1c2559a700 1 -- 172.21.15.77:0/414869190 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c1c004160 con 0x7f1c28108750 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.647+0000 7f1c2559a700 1 -- 172.21.15.77:0/414869190 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1c1c004300 con 0x7f1c28108750 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2559a700 1 -- 172.21.15.77:0/414869190 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c1c011a80 con 0x7f1c28108750 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2cd41700 1 -- 172.21.15.77:0/414869190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 msgr2=0x7f1c2810ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2cd41700 1 --2- 172.21.15.77:0/414869190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2810ab40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0098a0 tx=0x7f1c1c009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:11.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2cd41700 1 -- 172.21.15.77:0/414869190 shutdown_connections 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2cd41700 1 --2- 172.21.15.77:0/414869190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2810ab40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2cd41700 1 -- 172.21.15.77:0/414869190 >> 172.21.15.77:0/414869190 conn(0x7f1c28102180 msgr2=0x7f1c281045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2cd41700 1 -- 172.21.15.77:0/414869190 shutdown_connections 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.648+0000 7f1c2cd41700 1 -- 172.21.15.77:0/414869190 wait complete. 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.649+0000 7f1c2cd41700 1 Processor -- start 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.649+0000 7f1c2cd41700 1 -- start start 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.649+0000 7f1c2cd41700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.649+0000 7f1c2cd41700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2810bf60 con 0x7f1c28108750 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.649+0000 7f1c2659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.649+0000 7f1c2659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2819bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40460/0 (socket says 172.21.15.77:40460) 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.649+0000 7f1c2659c700 1 -- 172.21.15.77:0/3103351222 learned_addr learned my addr 172.21.15.77:0/3103351222 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c2659c700 1 -- 172.21.15.77:0/3103351222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2819c000 con 0x7f1c28108750 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c2659c700 1 --2- 172.21.15.77:0/3103351222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2819bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c006af0 tx=0x7f1c1c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:11.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c177fe700 1 -- 172.21.15.77:0/3103351222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c1c003930 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c177fe700 1 -- 172.21.15.77:0/3103351222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1c1c003ad0 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2819c290 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c177fe700 1 -- 172.21.15.77:0/3103351222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c1c0195e0 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c2819ef80 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c177fe700 1 -- 172.21.15.77:0/3103351222 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1c1c019c90 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.650+0000 7f1c177fe700 1 -- 172.21.15.77:0/3103351222 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1c1c00b880 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.651+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c28063900 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.653+0000 7f1c177fe700 1 -- 172.21.15.77:0/3103351222 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1c1c032070 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.725+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1c2819f520 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.726+0000 7f1c177fe700 1 -- 172.21.15.77:0/3103351222 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1c1c00ba80 con 0x7f1c28108750 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.727+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 msgr2=0x7f1c2819bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.727+0000 7f1c2cd41700 1 --2- 172.21.15.77:0/3103351222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2819bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c006af0 tx=0x7f1c1c011b70 comp rx=0 tx=0).stop 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.727+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 shutdown_connections 2024-03-16T09:32:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.727+0000 7f1c2cd41700 1 --2- 172.21.15.77:0/3103351222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c28108750 0x7f1c2819bac0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:11.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.727+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 >> 172.21.15.77:0/3103351222 conn(0x7f1c28102180 msgr2=0x7f1c28102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:11.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.727+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 shutdown_connections 2024-03-16T09:32:11.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:10.727+0000 7f1c2cd41700 1 -- 172.21.15.77:0/3103351222 wait complete. 2024-03-16T09:32:11.368 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (1/15)... 2024-03-16T09:32:11.619 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:11.384+0000 7f9e08d42200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T09:32:11.619 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:11.476+0000 7f9e08d42200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T09:32:11.619 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:11.616+0000 7f9e08d42200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T09:32:12.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:11.710+0000 7f9e08d42200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T09:32:12.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:11.791+0000 7f9e08d42200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T09:32:12.355 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:12.051+0000 7f9e08d42200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T09:32:12.355 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:12.159+0000 7f9e08d42200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T09:32:12.355 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:12.255+0000 7f9e08d42200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T09:32:12.355 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:12.352+0000 7f9e08d42200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T09:32:12.759 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:12.524+0000 7f9e08d42200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T09:32:13.765 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:13 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:13.509+0000 7f9e08d42200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T09:32:13.765 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:13 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:13.655+0000 7f9e08d42200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T09:32:14.030 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:13 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3193328206' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T09:32:14.326 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:14.027+0000 7f9e08d42200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T09:32:14.326 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:14.121+0000 7f9e08d42200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T09:32:14.326 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:14.323+0000 7f9e08d42200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T09:32:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T09:32:14.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T09:32:14.433 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T09:32:14.433 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T09:32:14.433 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T09:32:14.433 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:14.434 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:14.435 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T09:32:02.879942+0000", 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.780+0000 7f601080a700 1 Processor -- start 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.780+0000 7f601080a700 1 -- start start 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f601080a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f601080a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f600c0635d0 con 0x7f600c108780 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f600a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f600a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40476/0 (socket says 172.21.15.77:40476) 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f600a59c700 1 -- 172.21.15.77:0/3518335451 learned_addr learned my addr 172.21.15.77:0/3518335451 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f600a59c700 1 -- 172.21.15.77:0/3518335451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c10b0b0 con 0x7f600c108780 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f600a59c700 1 --2- 172.21.15.77:0/3518335451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c10ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0098a0 tx=0x7f5ffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce0fae8d97124ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.781+0000 7f600959a700 1 -- 172.21.15.77:0/3518335451 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ffc004160 con 0x7f600c108780 2024-03-16T09:32:14.436 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f600959a700 1 -- 172.21.15.77:0/3518335451 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5ffc004300 con 0x7f600c108780 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f600959a700 1 -- 172.21.15.77:0/3518335451 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ffc011a80 con 0x7f600c108780 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f601080a700 1 -- 172.21.15.77:0/3518335451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 msgr2=0x7f600c10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f601080a700 1 --2- 172.21.15.77:0/3518335451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c10ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0098a0 tx=0x7f5ffc009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f601080a700 1 -- 172.21.15.77:0/3518335451 shutdown_connections 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f601080a700 1 --2- 172.21.15.77:0/3518335451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c10ab70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f601080a700 1 -- 172.21.15.77:0/3518335451 >> 172.21.15.77:0/3518335451 conn(0x7f600c1021b0 msgr2=0x7f600c1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f601080a700 1 -- 172.21.15.77:0/3518335451 shutdown_connections 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.782+0000 7f601080a700 1 -- 172.21.15.77:0/3518335451 wait complete. 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.783+0000 7f601080a700 1 Processor -- start 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.783+0000 7f601080a700 1 -- start start 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f601080a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f601080a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f600c10bf90 con 0x7f600c108780 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f600a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f600a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c19bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40488/0 (socket says 172.21.15.77:40488) 2024-03-16T09:32:14.437 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f600a59c700 1 -- 172.21.15.77:0/3193328206 learned_addr learned my addr 172.21.15.77:0/3193328206 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f600a59c700 1 -- 172.21.15.77:0/3193328206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c19bff0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f600a59c700 1 --2- 172.21.15.77:0/3193328206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c19bab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc011a50 tx=0x7f5ffc004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f5ffb7fe700 1 -- 172.21.15.77:0/3193328206 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ffc0039f0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.784+0000 7f5ffb7fe700 1 -- 172.21.15.77:0/3193328206 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5ffc003b90 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.785+0000 7f601080a700 1 -- 172.21.15.77:0/3193328206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600c19c2e0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.785+0000 7f5ffb7fe700 1 -- 172.21.15.77:0/3193328206 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ffc01a5a0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.785+0000 7f601080a700 1 -- 172.21.15.77:0/3193328206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f600c19ef70 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.785+0000 7f5ffb7fe700 1 -- 172.21.15.77:0/3193328206 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ffc01ac60 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.785+0000 7f5ffb7fe700 1 -- 172.21.15.77:0/3193328206 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ffc0247d0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.787+0000 7f5ff97fa700 1 -- 172.21.15.77:0/3193328206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff0005330 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.789+0000 7f5ffb7fe700 1 -- 172.21.15.77:0/3193328206 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5ffc0249d0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.862+0000 7f5ff97fa700 1 -- 172.21.15.77:0/3193328206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5ff00058f0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.863+0000 7f5ffb7fe700 1 -- 172.21.15.77:0/3193328206 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5ffc00b6b0 con 0x7f600c108780 2024-03-16T09:32:14.438 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.864+0000 7f5ff97fa700 1 -- 172.21.15.77:0/3193328206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 msgr2=0x7f600c19bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:14.439 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.864+0000 7f5ff97fa700 1 --2- 172.21.15.77:0/3193328206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c19bab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc011a50 tx=0x7f5ffc004290 comp rx=0 tx=0).stop 2024-03-16T09:32:14.439 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.864+0000 7f5ff97fa700 1 -- 172.21.15.77:0/3193328206 shutdown_connections 2024-03-16T09:32:14.439 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.864+0000 7f5ff97fa700 1 --2- 172.21.15.77:0/3193328206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f600c108780 0x7f600c19bab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:14.439 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.864+0000 7f5ff97fa700 1 -- 172.21.15.77:0/3193328206 >> 172.21.15.77:0/3193328206 conn(0x7f600c1021b0 msgr2=0x7f600c102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:14.439 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.864+0000 7f5ff97fa700 1 -- 172.21.15.77:0/3193328206 shutdown_connections 2024-03-16T09:32:14.439 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:13.864+0000 7f5ff97fa700 1 -- 172.21.15.77:0/3193328206 wait complete. 2024-03-16T09:32:14.439 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (2/15)... 2024-03-16T09:32:14.759 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:14.520+0000 7f9e08d42200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T09:32:15.067 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:14.807+0000 7f9e08d42200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:32:15.067 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:15 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:15.064+0000 7f9e08d42200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T09:32:15.509 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:15 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:15.298+0000 7f9e08d42200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T09:32:16.509 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:16.094+0000 7f9e08d42200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T09:32:16.820 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:16.669+0000 7f9e08d42200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T09:32:16.820 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:16.769+0000 7f9e08d42200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T09:32:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:16 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1645092056' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T09:32:17.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:17.000+0000 7f9e08d42200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T09:32:17.627 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T09:32:17.629 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T09:32:17.630 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T09:32:02.879942+0000", 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:17.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.824+0000 7f083ac51700 1 Processor -- start 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.825+0000 7f083ac51700 1 -- start start 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.825+0000 7f083ac51700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f08341080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.825+0000 7f083ac51700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0834108690 con 0x7f0834107ca0 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.825+0000 7f08389ed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f08341080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.825+0000 7f08389ed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f08341080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55418/0 (socket says 172.21.15.77:55418) 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.825+0000 7f08389ed700 1 -- 172.21.15.77:0/1134149791 learned_addr learned my addr 172.21.15.77:0/1134149791 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.825+0000 7f08389ed700 1 -- 172.21.15.77:0/1134149791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0834108ea0 con 0x7f0834107ca0 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.826+0000 7f08389ed700 1 --2- 172.21.15.77:0/1134149791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f08341080c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08280098a0 tx=0x7f0828009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d5dc6a419f4cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.826+0000 7f08337fe700 1 -- 172.21.15.77:0/1134149791 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0828004160 con 0x7f0834107ca0 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.826+0000 7f08337fe700 1 -- 172.21.15.77:0/1134149791 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0828004300 con 0x7f0834107ca0 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.826+0000 7f08337fe700 1 -- 172.21.15.77:0/1134149791 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0828011a80 con 0x7f0834107ca0 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.826+0000 7f083ac51700 1 -- 172.21.15.77:0/1134149791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 msgr2=0x7f08341080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.826+0000 7f083ac51700 1 --2- 172.21.15.77:0/1134149791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f08341080c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08280098a0 tx=0x7f0828009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 -- 172.21.15.77:0/1134149791 shutdown_connections 2024-03-16T09:32:17.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 --2- 172.21.15.77:0/1134149791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f08341080c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 -- 172.21.15.77:0/1134149791 >> 172.21.15.77:0/1134149791 conn(0x7f0834103300 msgr2=0x7f0834105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 -- 172.21.15.77:0/1134149791 shutdown_connections 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 -- 172.21.15.77:0/1134149791 wait complete. 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 Processor -- start 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 -- start start 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.827+0000 7f083ac51700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f083419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f083ac51700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f083410c0d0 con 0x7f0834107ca0 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f08389ed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f083419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f08389ed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f083419bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55430/0 (socket says 172.21.15.77:55430) 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f08389ed700 1 -- 172.21.15.77:0/1645092056 learned_addr learned my addr 172.21.15.77:0/1645092056 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f08389ed700 1 -- 172.21.15.77:0/1645092056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083419bff0 con 0x7f0834107ca0 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f08389ed700 1 --2- 172.21.15.77:0/1645092056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f083419bab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0828006af0 tx=0x7f0828011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f0831ffb700 1 -- 172.21.15.77:0/1645092056 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08280039e0 con 0x7f0834107ca0 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f0831ffb700 1 -- 172.21.15.77:0/1645092056 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0828003b80 con 0x7f0834107ca0 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083419c280 con 0x7f0834107ca0 2024-03-16T09:32:17.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f0831ffb700 1 -- 172.21.15.77:0/1645092056 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08280196a0 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.828+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f083419ef70 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.829+0000 7f0831ffb700 1 -- 172.21.15.77:0/1645092056 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0828019ce0 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.829+0000 7f0831ffb700 1 -- 172.21.15.77:0/1645092056 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0828024730 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.829+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0834063900 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.831+0000 7f0831ffb700 1 -- 172.21.15.77:0/1645092056 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0828043dd0 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.903+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f083419f140 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.904+0000 7f0831ffb700 1 -- 172.21.15.77:0/1645092056 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0828032020 con 0x7f0834107ca0 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.905+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 msgr2=0x7f083419bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.905+0000 7f083ac51700 1 --2- 172.21.15.77:0/1645092056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f083419bab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0828006af0 tx=0x7f0828011c40 comp rx=0 tx=0).stop 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.905+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 shutdown_connections 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.905+0000 7f083ac51700 1 --2- 172.21.15.77:0/1645092056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0834107ca0 0x7f083419bab0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.905+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 >> 172.21.15.77:0/1645092056 conn(0x7f0834103300 msgr2=0x7f0834103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.905+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 shutdown_connections 2024-03-16T09:32:17.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:16.905+0000 7f083ac51700 1 -- 172.21.15.77:0/1645092056 wait complete. 2024-03-16T09:32:17.635 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (3/15)... 2024-03-16T09:32:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: Activating manager daemon x 2024-03-16T09:32:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: mgrmap e2: x(active, starting, since 0.00289396s) 2024-03-16T09:32:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T09:32:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T09:32:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T09:32:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:32:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T09:32:18.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: Manager daemon x is now available 2024-03-16T09:32:18.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T09:32:18.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T09:32:18.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' 2024-03-16T09:32:18.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' 2024-03-16T09:32:18.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:17 smithi077 ceph-mon[32171]: from='mgr.14100 172.21.15.77:0/4192591418' entity='mgr.x' 2024-03-16T09:32:19.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:19 smithi077 ceph-mon[32171]: mgrmap e3: x(active, since 1.00612s) 2024-03-16T09:32:20.818 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:20.818 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:20.818 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:32:20.818 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T09:32:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T09:32:20.820 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T09:32:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T09:32:20.822 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T09:32:02.879942+0000", 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.022+0000 7f8ef6bc9700 1 Processor -- start 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.022+0000 7f8ef6bc9700 1 -- start start 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.022+0000 7f8ef6bc9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef0107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.022+0000 7f8ef6bc9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef0107610 con 0x7f8ef0106c20 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.022+0000 7f8ef4965700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef0107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.022+0000 7f8ef4965700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef0107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55488/0 (socket says 172.21.15.77:55488) 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.022+0000 7f8ef4965700 1 -- 172.21.15.77:0/1948302555 learned_addr learned my addr 172.21.15.77:0/1948302555 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.023+0000 7f8ef4965700 1 -- 172.21.15.77:0/1948302555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef0107e20 con 0x7f8ef0106c20 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.023+0000 7f8ef4965700 1 --2- 172.21.15.77:0/1948302555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef0107040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098a0 tx=0x7f8ee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb15508e7c691c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:20.823 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.023+0000 7f8eef7fe700 1 -- 172.21.15.77:0/1948302555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ee4004160 con 0x7f8ef0106c20 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.023+0000 7f8eef7fe700 1 -- 172.21.15.77:0/1948302555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8ee4004300 con 0x7f8ef0106c20 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.023+0000 7f8eef7fe700 1 -- 172.21.15.77:0/1948302555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ee4011a80 con 0x7f8ef0106c20 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/1948302555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 msgr2=0x7f8ef0107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 --2- 172.21.15.77:0/1948302555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef0107040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098a0 tx=0x7f8ee4009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/1948302555 shutdown_connections 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 --2- 172.21.15.77:0/1948302555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef0107040 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/1948302555 >> 172.21.15.77:0/1948302555 conn(0x7f8ef0102220 msgr2=0x7f8ef0104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/1948302555 shutdown_connections 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/1948302555 wait complete. 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 Processor -- start 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.024+0000 7f8ef6bc9700 1 -- start start 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef6bc9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef6bc9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef010b050 con 0x7f8ef0106c20 2024-03-16T09:32:20.824 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef4965700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef4965700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55496/0 (socket says 172.21.15.77:55496) 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef4965700 1 -- 172.21.15.77:0/532851303 learned_addr learned my addr 172.21.15.77:0/532851303 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef4965700 1 -- 172.21.15.77:0/532851303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef019af90 con 0x7f8ef0106c20 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef4965700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef019aa50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4000c00 tx=0x7f8ee4004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8eedffb700 1 -- 172.21.15.77:0/532851303 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ee40039c0 con 0x7f8ef0106c20 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8eedffb700 1 -- 172.21.15.77:0/532851303 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8ee4003b60 con 0x7f8ef0106c20 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef019b220 con 0x7f8ef0106c20 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8eedffb700 1 -- 172.21.15.77:0/532851303 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ee401a6f0 con 0x7f8ef0106c20 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.025+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef019df10 con 0x7f8ef0106c20 2024-03-16T09:32:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.026+0000 7f8eedffb700 1 -- 172.21.15.77:0/532851303 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f8ee401a9a0 con 0x7f8ef0106c20 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.026+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef0063900 con 0x7f8ef0106c20 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.026+0000 7f8eedffb700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f8ee0038ed0 0x7f8ee003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.026+0000 7f8eedffb700 1 -- 172.21.15.77:0/532851303 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8ee404d030 con 0x7f8ef0106c20 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.027+0000 7f8eeffff700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f8ee0038ed0 0x7f8ee003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.027+0000 7f8eeffff700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f8ee0038ed0 0x7f8ee003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8edc006fd0 tx=0x7f8edc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.032+0000 7f8eedffb700 1 -- 172.21.15.77:0/532851303 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ee400bc40 con 0x7f8ef0106c20 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.305+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8ef019e0e0 con 0x7f8ef0106c20 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.306+0000 7f8eedffb700 1 -- 172.21.15.77:0/532851303 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8ef019e0e0 con 0x7f8ef0106c20 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f8ee0038ed0 msgr2=0x7f8ee003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f8ee0038ed0 0x7f8ee003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8edc006fd0 tx=0x7f8edc006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 msgr2=0x7f8ef019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef019aa50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4000c00 tx=0x7f8ee4004290 comp rx=0 tx=0).stop 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 shutdown_connections 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f8ee0038ed0 0x7f8ee003b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:20.826 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 --2- 172.21.15.77:0/532851303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ef0106c20 0x7f8ef019aa50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:20.827 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 >> 172.21.15.77:0/532851303 conn(0x7f8ef0102220 msgr2=0x7f8ef0102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:20.827 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 shutdown_connections 2024-03-16T09:32:20.827 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:20.308+0000 7f8ef6bc9700 1 -- 172.21.15.77:0/532851303 wait complete. 2024-03-16T09:32:20.827 INFO:teuthology.orchestra.run.smithi077.stdout:mgr is available 2024-03-16T09:32:21.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:21 smithi077 ceph-mon[32171]: mgrmap e4: x(active, since 3s) 2024-03-16T09:32:21.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:21 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/532851303' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T09:32:22.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:22 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3242577237' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout fsid = e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T09:32:22.171 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.351+0000 7f37d28b7700 1 Processor -- start 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.351+0000 7f37d28b7700 1 -- start start 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37d28b7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc107010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37d28b7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc1075e0 con 0x7f37cc106bf0 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55512/0 (socket says 172.21.15.77:55512) 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37cbfff700 1 -- 172.21.15.77:0/1137081434 learned_addr learned my addr 172.21.15.77:0/1137081434 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:22.172 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37cbfff700 1 -- 172.21.15.77:0/1137081434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37cc107df0 con 0x7f37cc106bf0 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37cbfff700 1 --2- 172.21.15.77:0/1137081434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc107010 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f37c00098a0 tx=0x7f37c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf4472455060571 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37caffd700 1 -- 172.21.15.77:0/1137081434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c0004160 con 0x7f37cc106bf0 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37caffd700 1 -- 172.21.15.77:0/1137081434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f37c0004300 con 0x7f37cc106bf0 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.352+0000 7f37caffd700 1 -- 172.21.15.77:0/1137081434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c0011a80 con 0x7f37cc106bf0 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 -- 172.21.15.77:0/1137081434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 msgr2=0x7f37cc107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 --2- 172.21.15.77:0/1137081434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc107010 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f37c00098a0 tx=0x7f37c0009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 -- 172.21.15.77:0/1137081434 shutdown_connections 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 --2- 172.21.15.77:0/1137081434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc107010 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 -- 172.21.15.77:0/1137081434 >> 172.21.15.77:0/1137081434 conn(0x7f37cc1021b0 msgr2=0x7f37cc1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 -- 172.21.15.77:0/1137081434 shutdown_connections 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 -- 172.21.15.77:0/1137081434 wait complete. 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 Processor -- start 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.353+0000 7f37d28b7700 1 -- start start 2024-03-16T09:32:22.173 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37d28b7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37d28b7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc10b020 con 0x7f37cc106bf0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc19aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55528/0 (socket says 172.21.15.77:55528) 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37cbfff700 1 -- 172.21.15.77:0/3242577237 learned_addr learned my addr 172.21.15.77:0/3242577237 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37cbfff700 1 -- 172.21.15.77:0/3242577237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37cc19afb0 con 0x7f37cc106bf0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37cbfff700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc19aa70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37c0000c00 tx=0x7f37c0004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37c97fa700 1 -- 172.21.15.77:0/3242577237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c00039c0 con 0x7f37cc106bf0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37c97fa700 1 -- 172.21.15.77:0/3242577237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f37c0003b60 con 0x7f37cc106bf0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37c97fa700 1 -- 172.21.15.77:0/3242577237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c001a6f0 con 0x7f37cc106bf0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.354+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37cc19b240 con 0x7f37cc106bf0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.355+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37cc19df30 con 0x7f37cc106bf0 2024-03-16T09:32:22.174 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.355+0000 7f37c97fa700 1 -- 172.21.15.77:0/3242577237 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f37c001a890 con 0x7f37cc106bf0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.355+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37cc063900 con 0x7f37cc106bf0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.355+0000 7f37c97fa700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f37b4038f20 0x7f37b403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.356+0000 7f37c97fa700 1 -- 172.21.15.77:0/3242577237 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f37c004cfe0 con 0x7f37cc106bf0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.356+0000 7f37cb7fe700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f37b4038f20 0x7f37b403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.356+0000 7f37cb7fe700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f37b4038f20 0x7f37b403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f37bc006fd0 tx=0x7f37bc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.360+0000 7f37c97fa700 1 -- 172.21.15.77:0/3242577237 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37c000bc10 con 0x7f37cc106bf0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.556+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f37cc19e100 con 0x7f37cc106bf0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.557+0000 7f37c97fa700 1 -- 172.21.15.77:0/3242577237 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+400 (secure 0 0 0) 0x7f37cc19e100 con 0x7f37cc106bf0 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.559+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f37b4038f20 msgr2=0x7f37b403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.559+0000 7f37d28b7700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f37b4038f20 0x7f37b403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f37bc006fd0 tx=0x7f37bc006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.559+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 msgr2=0x7f37cc19aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.559+0000 7f37d28b7700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc19aa70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37c0000c00 tx=0x7f37c0004290 comp rx=0 tx=0).stop 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.560+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 shutdown_connections 2024-03-16T09:32:22.175 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.560+0000 7f37d28b7700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f37b4038f20 0x7f37b403b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:22.176 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.560+0000 7f37d28b7700 1 --2- 172.21.15.77:0/3242577237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37cc106bf0 0x7f37cc19aa70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:22.176 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.560+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 >> 172.21.15.77:0/3242577237 conn(0x7f37cc1021b0 msgr2=0x7f37cc102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:22.176 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.560+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 shutdown_connections 2024-03-16T09:32:22.176 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:21.560+0000 7f37d28b7700 1 -- 172.21.15.77:0/3242577237 wait complete. 2024-03-16T09:32:22.176 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling cephadm module... 2024-03-16T09:32:23.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:23 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3656451001' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-16T09:32:23.400 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: ignoring --setuser ceph since I am not root 2024-03-16T09:32:23.400 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: ignoring --setgroup ceph since I am not root 2024-03-16T09:32:23.671 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:23.574+0000 7f3b35853200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T09:32:23.671 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:23.668+0000 7f3b35853200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T09:32:23.707 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.551+0000 7f37b32f2700 1 Processor -- start 2024-03-16T09:32:23.707 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.551+0000 7f37b32f2700 1 -- start start 2024-03-16T09:32:23.707 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.551+0000 7f37b32f2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:23.707 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.551+0000 7f37b32f2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac07c870 con 0x7f37ac07de40 2024-03-16T09:32:23.707 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.551+0000 7f37b108e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac07c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:23.707 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.551+0000 7f37b108e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac07c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55530/0 (socket says 172.21.15.77:55530) 2024-03-16T09:32:23.707 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.551+0000 7f37b108e700 1 -- 172.21.15.77:0/2955346391 learned_addr learned my addr 172.21.15.77:0/2955346391 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.552+0000 7f37b108e700 1 -- 172.21.15.77:0/2955346391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ac07c9f0 con 0x7f37ac07de40 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.552+0000 7f37b108e700 1 --2- 172.21.15.77:0/2955346391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac07c2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f37a80098a0 tx=0x7f37a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa59c4172d82033 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.552+0000 7f37a3fff700 1 -- 172.21.15.77:0/2955346391 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37a8004160 con 0x7f37ac07de40 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.552+0000 7f37a3fff700 1 -- 172.21.15.77:0/2955346391 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f37a8004300 con 0x7f37ac07de40 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.552+0000 7f37a3fff700 1 -- 172.21.15.77:0/2955346391 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37a8011a80 con 0x7f37ac07de40 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 -- 172.21.15.77:0/2955346391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 msgr2=0x7f37ac07c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 --2- 172.21.15.77:0/2955346391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac07c2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f37a80098a0 tx=0x7f37a8009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 -- 172.21.15.77:0/2955346391 shutdown_connections 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 --2- 172.21.15.77:0/2955346391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac07c2a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 -- 172.21.15.77:0/2955346391 >> 172.21.15.77:0/2955346391 conn(0x7f37ac1032c0 msgr2=0x7f37ac1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 -- 172.21.15.77:0/2955346391 shutdown_connections 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 -- 172.21.15.77:0/2955346391 wait complete. 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 Processor -- start 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 -- start start 2024-03-16T09:32:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.553+0000 7f37b32f2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac10e180 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37b108e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37b108e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55540/0 (socket says 172.21.15.77:55540) 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37b108e700 1 -- 172.21.15.77:0/3656451001 learned_addr learned my addr 172.21.15.77:0/3656451001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37b108e700 1 -- 172.21.15.77:0/3656451001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ac19e250 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37b108e700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac19dd10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f37a8006af0 tx=0x7f37a8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37a80039e0 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f37a8003b80 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ac19e4e0 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37a80196a0 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.554+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ac1a11c0 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.555+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f37a80199a0 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.555+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ac063900 con 0x7f37ac07de40 2024-03-16T09:32:23.709 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.555+0000 7f37a27fc700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f3798038e80 0x7f379803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.555+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f37a804d510 con 0x7f37ac07de40 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.555+0000 7f37b088d700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f3798038e80 0x7f379803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.556+0000 7f37b088d700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f3798038e80 0x7f379803b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f379c006fd0 tx=0x7f379c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.560+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37a804cf30 con 0x7f37ac07de40 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:22.809+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f37ac1a1390 con 0x7f37ac07de40 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.021+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f37ac1a1390 con 0x7f37ac07de40 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.023+0000 7f37a27fc700 1 -- 172.21.15.77:0/3656451001 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f37a802d440 con 0x7f37ac07de40 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f3798038e80 msgr2=0x7f379803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f3798038e80 0x7f379803b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f379c006fd0 tx=0x7f379c006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 msgr2=0x7f37ac19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac19dd10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f37a8006af0 tx=0x7f37a8011c40 comp rx=0 tx=0).stop 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 shutdown_connections 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f3798038e80 0x7f379803b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:23.710 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 --2- 172.21.15.77:0/3656451001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37ac07de40 0x7f37ac19dd10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:23.711 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 >> 172.21.15.77:0/3656451001 conn(0x7f37ac1032c0 msgr2=0x7f37ac103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:23.711 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 shutdown_connections 2024-03-16T09:32:23.711 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:23.024+0000 7f37b32f2700 1 -- 172.21.15.77:0/3656451001 wait complete. 2024-03-16T09:32:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:24 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3656451001' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-16T09:32:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:24 smithi077 ceph-mon[32171]: mgrmap e5: x(active, since 6s) 2024-03-16T09:32:24.592 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:24 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:24.323+0000 7f3b35853200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T09:32:24.593 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:24 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:24.419+0000 7f3b35853200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T09:32:24.593 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:24 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:24.562+0000 7f3b35853200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T09:32:25.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:24 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:24.666+0000 7f3b35853200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T09:32:25.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:24 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:24.749+0000 7f3b35853200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T09:32:25.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:25.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-16T09:32:25.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T09:32:25.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T09:32:25.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T09:32:25.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:25.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.124+0000 7f1d3679f700 1 Processor -- start 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.124+0000 7f1d3679f700 1 -- start start 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d3679f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d30106e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d3679f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301073d0 con 0x7f1d301069e0 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d3579d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d30106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d3579d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d30106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55556/0 (socket says 172.21.15.77:55556) 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d3579d700 1 -- 172.21.15.77:0/2927941496 learned_addr learned my addr 172.21.15.77:0/2927941496 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d3579d700 1 -- 172.21.15.77:0/2927941496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d30107be0 con 0x7f1d301069e0 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d3579d700 1 --2- 172.21.15.77:0/2927941496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d30106e00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00bb40 tx=0x7f1d2c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48143c7b30cdbd16 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d27fff700 1 -- 172.21.15.77:0/2927941496 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d2c00dbe0 con 0x7f1d301069e0 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d27fff700 1 -- 172.21.15.77:0/2927941496 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1d2c0044c0 con 0x7f1d301069e0 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.125+0000 7f1d27fff700 1 -- 172.21.15.77:0/2927941496 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d2c010470 con 0x7f1d301069e0 2024-03-16T09:32:25.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.126+0000 7f1d3679f700 1 -- 172.21.15.77:0/2927941496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 msgr2=0x7f1d30106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.126+0000 7f1d3679f700 1 --2- 172.21.15.77:0/2927941496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d30106e00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00bb40 tx=0x7f1d2c00be50 comp rx=0 tx=0).stop 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.126+0000 7f1d3679f700 1 -- 172.21.15.77:0/2927941496 shutdown_connections 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.126+0000 7f1d3679f700 1 --2- 172.21.15.77:0/2927941496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d30106e00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.126+0000 7f1d3679f700 1 -- 172.21.15.77:0/2927941496 >> 172.21.15.77:0/2927941496 conn(0x7f1d30101f80 msgr2=0x7f1d301043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.127+0000 7f1d3679f700 1 -- 172.21.15.77:0/2927941496 shutdown_connections 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.127+0000 7f1d3679f700 1 -- 172.21.15.77:0/2927941496 wait complete. 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.127+0000 7f1d3679f700 1 Processor -- start 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.127+0000 7f1d3679f700 1 -- start start 2024-03-16T09:32:25.067 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.127+0000 7f1d3679f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d3019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.127+0000 7f1d3679f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30108a70 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d3579d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d3019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d3579d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d3019dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55564/0 (socket says 172.21.15.77:55564) 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d3579d700 1 -- 172.21.15.77:0/255996725 learned_addr learned my addr 172.21.15.77:0/255996725 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d3579d700 1 -- 172.21.15.77:0/255996725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3019e0f0 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d3579d700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d3019dbb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00bfd0 tx=0x7f1d2c003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d267fc700 1 -- 172.21.15.77:0/255996725 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d2c010770 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d267fc700 1 -- 172.21.15.77:0/255996725 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1d2c01d070 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3019e380 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d267fc700 1 -- 172.21.15.77:0/255996725 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d2c019bd0 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.128+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d301a1070 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.129+0000 7f1d267fc700 1 -- 172.21.15.77:0/255996725 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f1d2c028060 con 0x7f1d301069e0 2024-03-16T09:32:25.068 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.129+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d30063900 con 0x7f1d301069e0 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.130+0000 7f1d267fc700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 0x7f1d1c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.130+0000 7f1d267fc700 1 -- 172.21.15.77:0/255996725 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1d2c04d0b0 con 0x7f1d301069e0 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.130+0000 7f1d34f9c700 1 -- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 msgr2=0x7f1d1c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/262452209 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.130+0000 7f1d34f9c700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 0x7f1d1c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.134+0000 7f1d267fc700 1 -- 172.21.15.77:0/255996725 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1d2c015070 con 0x7f1d301069e0 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.331+0000 7f1d34f9c700 1 -- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 msgr2=0x7f1d1c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/262452209 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.331+0000 7f1d34f9c700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 0x7f1d1c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.390+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f1d301a1290 con 0x7f1d301069e0 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.390+0000 7f1d267fc700 1 -- 172.21.15.77:0/255996725 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f1d301a1290 con 0x7f1d301069e0 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 msgr2=0x7f1d1c03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 0x7f1d1c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 msgr2=0x7f1d3019dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d3019dbb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00bfd0 tx=0x7f1d2c003fe0 comp rx=0 tx=0).stop 2024-03-16T09:32:25.069 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 shutdown_connections 2024-03-16T09:32:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7f1d1c038f20 0x7f1d1c03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 --2- 172.21.15.77:0/255996725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d301069e0 0x7f1d3019dbb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.392+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 >> 172.21.15.77:0/255996725 conn(0x7f1d30101f80 msgr2=0x7f1d3018fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.393+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 shutdown_connections 2024-03-16T09:32:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:24.393+0000 7f1d3679f700 1 -- 172.21.15.77:0/255996725 wait complete. 2024-03-16T09:32:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for the mgr to restart... 2024-03-16T09:32:25.070 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr epoch 5... 2024-03-16T09:32:25.313 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:25 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/255996725' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T09:32:25.313 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:25 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:25.012+0000 7f3b35853200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T09:32:25.313 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:25 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:25.131+0000 7f3b35853200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T09:32:25.313 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:25 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:25.216+0000 7f3b35853200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T09:32:25.313 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:25 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:25.310+0000 7f3b35853200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T09:32:25.740 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:25 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:25.489+0000 7f3b35853200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T09:32:26.759 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:26 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:26.433+0000 7f3b35853200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T09:32:26.759 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:26 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:26.560+0000 7f3b35853200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T09:32:27.169 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:26 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:26.886+0000 7f3b35853200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T09:32:27.169 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:26 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:26.974+0000 7f3b35853200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T09:32:27.169 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:27 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:27.166+0000 7f3b35853200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:32:27.509 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:27 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:27.347+0000 7f3b35853200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T09:32:27.879 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:27 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:27.625+0000 7f3b35853200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:32:28.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:27 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:27.877+0000 7f3b35853200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T09:32:28.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:28 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:28.109+0000 7f3b35853200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T09:32:29.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:28 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:28.894+0000 7f3b35853200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T09:32:29.759 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:29.452+0000 7f3b35853200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T09:32:29.759 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:29.548+0000 7f3b35853200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:29.758+0000 7f3b35853200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: Active manager daemon x restarted 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: Activating manager daemon x 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: osdmap e2: 0 total, 0 up, 0 in 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: mgrmap e6: x(active, starting, since 0.00343307s) 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T09:32:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: Manager daemon x is now available 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:29 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:31.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:30 smithi077 ceph-mon[32171]: Found migration_current of "None". Setting to last migration. 2024-03-16T09:32:31.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:30 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T09:32:31.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:30 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T09:32:31.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:30 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:31.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:30 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:31.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:30 smithi077 ceph-mon[32171]: mgrmap e7: x(active, since 1.00585s) 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.531+0000 7fb8074f4700 1 Processor -- start 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.531+0000 7fb8074f4700 1 -- start start 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb8074f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb8001069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:31.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb8074f4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb80007bdb0 con 0x7fb8001065b0 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb805290700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb8001069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb805290700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb8001069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57772/0 (socket says 172.21.15.77:57772) 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb805290700 1 -- 172.21.15.77:0/3327249847 learned_addr learned my addr 172.21.15.77:0/3327249847 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb805290700 1 -- 172.21.15.77:0/3327249847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb800106f10 con 0x7fb8001065b0 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb805290700 1 --2- 172.21.15.77:0/3327249847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb8001069d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0098a0 tx=0x7fb7fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4cad75feeeed6f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb7f7fff700 1 -- 172.21.15.77:0/3327249847 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc004160 con 0x7fb8001065b0 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb7f7fff700 1 -- 172.21.15.77:0/3327249847 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb7fc004300 con 0x7fb8001065b0 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.532+0000 7fb7f7fff700 1 -- 172.21.15.77:0/3327249847 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc011a80 con 0x7fb8001065b0 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.533+0000 7fb8074f4700 1 -- 172.21.15.77:0/3327249847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 msgr2=0x7fb8001069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.533+0000 7fb8074f4700 1 --2- 172.21.15.77:0/3327249847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb8001069d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0098a0 tx=0x7fb7fc009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.533+0000 7fb8074f4700 1 -- 172.21.15.77:0/3327249847 shutdown_connections 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.533+0000 7fb8074f4700 1 --2- 172.21.15.77:0/3327249847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb8001069d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.533+0000 7fb8074f4700 1 -- 172.21.15.77:0/3327249847 >> 172.21.15.77:0/3327249847 conn(0x7fb8001021d0 msgr2=0x7fb8001045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.533+0000 7fb8074f4700 1 -- 172.21.15.77:0/3327249847 shutdown_connections 2024-03-16T09:32:31.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.533+0000 7fb8074f4700 1 -- 172.21.15.77:0/3327249847 wait complete. 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.534+0000 7fb8074f4700 1 Processor -- start 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.534+0000 7fb8074f4700 1 -- start start 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.534+0000 7fb8074f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb80019dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.534+0000 7fb8074f4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb800107e60 con 0x7fb8001065b0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb805290700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb80019dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb805290700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb80019dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57782/0 (socket says 172.21.15.77:57782) 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb805290700 1 -- 172.21.15.77:0/1381666762 learned_addr learned my addr 172.21.15.77:0/1381666762 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb805290700 1 -- 172.21.15.77:0/1381666762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80019e210 con 0x7fb8001065b0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb805290700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb80019dcd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc006af0 tx=0x7fb7fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc0039e0 con 0x7fb8001065b0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80019e4a0 con 0x7fb8001065b0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb7fc003b80 con 0x7fb8001065b0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc0196a0 con 0x7fb8001065b0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.535+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8001a1180 con 0x7fb8001065b0 2024-03-16T09:32:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.536+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fb7fc0199a0 con 0x7fb8001065b0 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.536+0000 7fb7f67fc700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.536+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 --> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb7ec03baa0 con 0x7fb7ec038ed0 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.536+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb7fc04d5c0 con 0x7fb8001065b0 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.536+0000 7fb804a8f700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 msgr2=0x7fb7ec03b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/262452209 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.536+0000 7fb804a8f700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.737+0000 7fb804a8f700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 msgr2=0x7fb7ec03b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/262452209 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:25.737+0000 7fb804a8f700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:26.138+0000 7fb804a8f700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 msgr2=0x7fb7ec03b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/262452209 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:26.138+0000 7fb804a8f700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:26.939+0000 7fb804a8f700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 msgr2=0x7fb7ec03b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/262452209 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:26.939+0000 7fb804a8f700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:28.540+0000 7fb804a8f700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 msgr2=0x7fb7ec03b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/262452209 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:28.540+0000 7fb804a8f700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:29.773+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fb7fc02dce0 con 0x7fb8001065b0 2024-03-16T09:32:31.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:29.773+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 msgr2=0x7fb7ec03b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:29.773+0000 7fb7f67fc700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.774+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fb7fc04f290 con 0x7fb8001065b0 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.775+0000 7fb7f67fc700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fb7ec03c7c0 0x7fb7ec03ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.775+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb7fc010ec0 con 0x7fb7ec03c7c0 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.775+0000 7fb804a8f700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fb7ec03c7c0 0x7fb7ec03ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.776+0000 7fb804a8f700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fb7ec03c7c0 0x7fb7ec03ebb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0003a10 tx=0x7fb7f0009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.776+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fb7fc010ec0 con 0x7fb7ec03c7c0 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fb800063900 con 0x7fb7ec03c7c0 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb7f67fc700 1 -- 172.21.15.77:0/1381666762 <== mgr.14120 v2:172.21.15.77:6800/3741578273 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fb800063900 con 0x7fb7ec03c7c0 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fb7ec03c7c0 msgr2=0x7fb7ec03ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fb7ec03c7c0 0x7fb7ec03ebb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0003a10 tx=0x7fb7f0009290 comp rx=0 tx=0).stop 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 msgr2=0x7fb80019dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:31.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb80019dcd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc006af0 tx=0x7fb7fc011c40 comp rx=0 tx=0).stop 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 shutdown_connections 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fb7ec03c7c0 0x7fb7ec03ebb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8001065b0 0x7fb80019dcd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 --2- 172.21.15.77:0/1381666762 >> [v2:172.21.15.77:6800/262452209,v1:172.21.15.77:6801/262452209] conn(0x7fb7ec038ed0 0x7fb7ec03b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 >> 172.21.15.77:0/1381666762 conn(0x7fb8001021d0 msgr2=0x7fb800194970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 shutdown_connections 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:30.783+0000 7fb8074f4700 1 -- 172.21.15.77:0/1381666762 wait complete. 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:mgr epoch 5 is available 2024-03-16T09:32:31.371 INFO:teuthology.orchestra.run.smithi077.stdout:Setting orchestrator backend to cephadm... 2024-03-16T09:32:32.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:31 smithi077 ceph-mon[32171]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T09:32:32.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:31 smithi077 ceph-mon[32171]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T09:32:32.660 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f3065f6b700 1 Processor -- start 2024-03-16T09:32:32.660 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f3065f6b700 1 -- start start 2024-03-16T09:32:32.660 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f3065f6b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f306010ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:32.660 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f3065f6b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30600635d0 con 0x7f3060108780 2024-03-16T09:32:32.660 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f305f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f306010ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:32.660 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f305f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f306010ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57838/0 (socket says 172.21.15.77:57838) 2024-03-16T09:32:32.660 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f305f7fe700 1 -- 172.21.15.77:0/1146388426 learned_addr learned my addr 172.21.15.77:0/1146388426 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f305f7fe700 1 -- 172.21.15.77:0/1146388426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306010b0b0 con 0x7f3060108780 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.830+0000 7f305f7fe700 1 --2- 172.21.15.77:0/1146388426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f306010ab70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30500098a0 tx=0x7f3050009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba91e78c97d87a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f305e7fc700 1 -- 172.21.15.77:0/1146388426 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3050004160 con 0x7f3060108780 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f305e7fc700 1 -- 172.21.15.77:0/1146388426 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3050004300 con 0x7f3060108780 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f305e7fc700 1 -- 172.21.15.77:0/1146388426 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3050011a80 con 0x7f3060108780 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f3065f6b700 1 -- 172.21.15.77:0/1146388426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 msgr2=0x7f306010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f3065f6b700 1 --2- 172.21.15.77:0/1146388426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f306010ab70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30500098a0 tx=0x7f3050009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f3065f6b700 1 -- 172.21.15.77:0/1146388426 shutdown_connections 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f3065f6b700 1 --2- 172.21.15.77:0/1146388426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f306010ab70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f3065f6b700 1 -- 172.21.15.77:0/1146388426 >> 172.21.15.77:0/1146388426 conn(0x7f30601021b0 msgr2=0x7f30601045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f3065f6b700 1 -- 172.21.15.77:0/1146388426 shutdown_connections 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.831+0000 7f3065f6b700 1 -- 172.21.15.77:0/1146388426 wait complete. 2024-03-16T09:32:32.661 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.832+0000 7f3065f6b700 1 Processor -- start 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.832+0000 7f3065f6b700 1 -- start start 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.832+0000 7f3065f6b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f3060108000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.832+0000 7f3065f6b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f306010bf90 con 0x7f3060108780 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.832+0000 7f305f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f3060108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.832+0000 7f305f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f3060108000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57852/0 (socket says 172.21.15.77:57852) 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.832+0000 7f305f7fe700 1 -- 172.21.15.77:0/2647378518 learned_addr learned my addr 172.21.15.77:0/2647378518 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.833+0000 7f305f7fe700 1 -- 172.21.15.77:0/2647378518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3060108540 con 0x7f3060108780 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.833+0000 7f305f7fe700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f3060108000 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3050011a50 tx=0x7f3050004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.833+0000 7f305cff9700 1 -- 172.21.15.77:0/2647378518 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30500039f0 con 0x7f3060108780 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.833+0000 7f305cff9700 1 -- 172.21.15.77:0/2647378518 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3050003b90 con 0x7f3060108780 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.833+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3060106730 con 0x7f3060108780 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.833+0000 7f305cff9700 1 -- 172.21.15.77:0/2647378518 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f305001a600 con 0x7f3060108780 2024-03-16T09:32:32.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.833+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3060106c70 con 0x7f3060108780 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.834+0000 7f305cff9700 1 -- 172.21.15.77:0/2647378518 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f305001a8b0 con 0x7f3060108780 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.834+0000 7f305cff9700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3048038f70 0x7f304803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.834+0000 7f305cff9700 1 -- 172.21.15.77:0/2647378518 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f305004d450 con 0x7f3060108780 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.834+0000 7f305effd700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3048038f70 0x7f304803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.834+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3060063900 con 0x7f3060108780 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.835+0000 7f305effd700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3048038f70 0x7f304803b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3054006fd0 tx=0x7f3054006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:31.839+0000 7f305cff9700 1 -- 172.21.15.77:0/2647378518 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3050019c70 con 0x7f3060108780 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.058+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f30601073f0 con 0x7f3048038f70 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.064+0000 7f305cff9700 1 -- 172.21.15.77:0/2647378518 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30601073f0 con 0x7f3048038f70 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3048038f70 msgr2=0x7f304803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:32.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3048038f70 0x7f304803b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3054006fd0 tx=0x7f3054006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 msgr2=0x7f3060108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f3060108000 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3050011a50 tx=0x7f3050004290 comp rx=0 tx=0).stop 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 shutdown_connections 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3048038f70 0x7f304803b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 --2- 172.21.15.77:0/2647378518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3060108780 0x7f3060108000 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.066+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 >> 172.21.15.77:0/2647378518 conn(0x7f30601021b0 msgr2=0x7f3060102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.067+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 shutdown_connections 2024-03-16T09:32:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:32.067+0000 7f3065f6b700 1 -- 172.21.15.77:0/2647378518 wait complete. 2024-03-16T09:32:32.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:32 smithi077 ceph-mon[32171]: mgrmap e8: x(active, since 2s) 2024-03-16T09:32:32.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:32 smithi077 ceph-mon[32171]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:32:32.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:32 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:32.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:32 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:32.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:32 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:33.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:33 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:32] ENGINE Bus STARTING 2024-03-16T09:32:33.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:33 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:32] ENGINE Serving on http://172.21.15.77:8765 2024-03-16T09:32:33.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:33 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:32] ENGINE Serving on https://172.21.15.77:7150 2024-03-16T09:32:33.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:33 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:32] ENGINE Bus STARTED 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b42ab700 1 Processor -- start 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b42ab700 1 -- start start 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b42ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b42ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1086f0 con 0x7f13ac107d00 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b2047700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b2047700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57866/0 (socket says 172.21.15.77:57866) 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b2047700 1 -- 172.21.15.77:0/3912374256 learned_addr learned my addr 172.21.15.77:0/3912374256 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.074+0000 7f13b2047700 1 -- 172.21.15.77:0/3912374256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac108f00 con 0x7f13ac107d00 2024-03-16T09:32:33.908 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b2047700 1 --2- 172.21.15.77:0/3912374256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac108120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13a80098a0 tx=0x7f13a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4507f0e104901172 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b1045700 1 -- 172.21.15.77:0/3912374256 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13a8004160 con 0x7f13ac107d00 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b1045700 1 -- 172.21.15.77:0/3912374256 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f13a8004300 con 0x7f13ac107d00 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b1045700 1 -- 172.21.15.77:0/3912374256 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13a8011a80 con 0x7f13ac107d00 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b42ab700 1 -- 172.21.15.77:0/3912374256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 msgr2=0x7f13ac108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b42ab700 1 --2- 172.21.15.77:0/3912374256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac108120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13a80098a0 tx=0x7f13a8009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b42ab700 1 -- 172.21.15.77:0/3912374256 shutdown_connections 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b42ab700 1 --2- 172.21.15.77:0/3912374256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac108120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b42ab700 1 -- 172.21.15.77:0/3912374256 >> 172.21.15.77:0/3912374256 conn(0x7f13ac103340 msgr2=0x7f13ac105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b42ab700 1 -- 172.21.15.77:0/3912374256 shutdown_connections 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.075+0000 7f13b42ab700 1 -- 172.21.15.77:0/3912374256 wait complete. 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b42ab700 1 Processor -- start 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b42ab700 1 -- start start 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b42ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac07daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b42ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac10c130 con 0x7f13ac107d00 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b2047700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac07daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b2047700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac07daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57874/0 (socket says 172.21.15.77:57874) 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b2047700 1 -- 172.21.15.77:0/2593281570 learned_addr learned my addr 172.21.15.77:0/2593281570 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.076+0000 7f13b2047700 1 -- 172.21.15.77:0/2593281570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac07e030 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.077+0000 7f13b2047700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac07daf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006af0 tx=0x7f13a8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.077+0000 7f13a37fe700 1 -- 172.21.15.77:0/2593281570 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13a8003930 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.077+0000 7f13a37fe700 1 -- 172.21.15.77:0/2593281570 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f13a8003ad0 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.077+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac07c220 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.077+0000 7f13a37fe700 1 -- 172.21.15.77:0/2593281570 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13a80195e0 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.077+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac07c760 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.078+0000 7f13a37fe700 1 -- 172.21.15.77:0/2593281570 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f13a8019890 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.078+0000 7f13a37fe700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1398038f20 0x7f139803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.078+0000 7f13a37fe700 1 -- 172.21.15.77:0/2593281570 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f13a804d2e0 con 0x7f13ac107d00 2024-03-16T09:32:33.910 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.078+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ac063900 con 0x7f13ac107d00 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.078+0000 7f13b1846700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1398038f20 0x7f139803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.078+0000 7f13b1846700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1398038f20 0x7f139803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f139c006fd0 tx=0x7f139c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.083+0000 7f13a37fe700 1 -- 172.21.15.77:0/2593281570 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f13a800bc40 con 0x7f13ac107d00 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.279+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f13ac07d050 con 0x7f1398038f20 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.279+0000 7f13a37fe700 1 -- 172.21.15.77:0/2593281570 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f13ac07d050 con 0x7f1398038f20 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1398038f20 msgr2=0x7f139803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1398038f20 0x7f139803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f139c006fd0 tx=0x7f139c006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 msgr2=0x7f13ac07daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac07daf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006af0 tx=0x7f13a8011b70 comp rx=0 tx=0).stop 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 shutdown_connections 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1398038f20 0x7f139803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 --2- 172.21.15.77:0/2593281570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac107d00 0x7f13ac07daf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 >> 172.21.15.77:0/2593281570 conn(0x7f13ac103340 msgr2=0x7f13ac103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:33.911 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 shutdown_connections 2024-03-16T09:32:33.912 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:33.281+0000 7f13b42ab700 1 -- 172.21.15.77:0/2593281570 wait complete. 2024-03-16T09:32:33.912 INFO:teuthology.orchestra.run.smithi077.stdout:Generating ssh key... 2024-03-16T09:32:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:34 smithi077 ceph-mon[32171]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:32:35.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: Generating public/private rsa key pair. 2024-03-16T09:32:35.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: Your identification has been saved in /tmp/tmpg_dxmbwl/key. 2024-03-16T09:32:35.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: Your public key has been saved in /tmp/tmpg_dxmbwl/key.pub. 2024-03-16T09:32:35.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: The key fingerprint is: 2024-03-16T09:32:35.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: SHA256:pQAfPfJiMfT6E9WC7i+JEj44hWHBv9xniFDr2YgYtYs ceph-e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:35.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: The key's randomart image is: 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: +---[RSA 3072]----+ 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: | . ..o. | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: | + .o+oo. . | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: | . = .o=o.+ . | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: |. = o o+.+ . | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: | = O O.oS | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: |E + X +oo. | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: | + . ++. | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: | o + . oo | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: | . o .. | 2024-03-16T09:32:35.260 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: +----[SHA256]-----+ 2024-03-16T09:32:35.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.281+0000 7f3e6250f700 1 Processor -- start 2024-03-16T09:32:35.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.281+0000 7f3e6250f700 1 -- start start 2024-03-16T09:32:35.631 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.281+0000 7f3e6250f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.281+0000 7f3e6250f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c1086f0 con 0x7f3e5c107d00 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.281+0000 7f3e5bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.281+0000 7f3e5bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57880/0 (socket says 172.21.15.77:57880) 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.281+0000 7f3e5bfff700 1 -- 172.21.15.77:0/488181004 learned_addr learned my addr 172.21.15.77:0/488181004 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.282+0000 7f3e5bfff700 1 -- 172.21.15.77:0/488181004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c108f00 con 0x7f3e5c107d00 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.282+0000 7f3e5bfff700 1 --2- 172.21.15.77:0/488181004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c108120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0098a0 tx=0x7f3e4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6084048377e74a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.282+0000 7f3e5affd700 1 -- 172.21.15.77:0/488181004 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e4c004160 con 0x7f3e5c107d00 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.282+0000 7f3e5affd700 1 -- 172.21.15.77:0/488181004 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3e4c004300 con 0x7f3e5c107d00 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.282+0000 7f3e5affd700 1 -- 172.21.15.77:0/488181004 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e4c011a80 con 0x7f3e5c107d00 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 -- 172.21.15.77:0/488181004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 msgr2=0x7f3e5c108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 --2- 172.21.15.77:0/488181004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c108120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0098a0 tx=0x7f3e4c009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 -- 172.21.15.77:0/488181004 shutdown_connections 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 --2- 172.21.15.77:0/488181004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c108120 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 -- 172.21.15.77:0/488181004 >> 172.21.15.77:0/488181004 conn(0x7f3e5c103340 msgr2=0x7f3e5c105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 -- 172.21.15.77:0/488181004 shutdown_connections 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 -- 172.21.15.77:0/488181004 wait complete. 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 Processor -- start 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.283+0000 7f3e6250f700 1 -- start start 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e6250f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c19bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e6250f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c10c130 con 0x7f3e5c107d00 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e5bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c19bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e5bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c19bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57894/0 (socket says 172.21.15.77:57894) 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e5bfff700 1 -- 172.21.15.77:0/608345339 learned_addr learned my addr 172.21.15.77:0/608345339 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e5bfff700 1 -- 172.21.15.77:0/608345339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c19c0b0 con 0x7f3e5c107d00 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e5bfff700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c19bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006af0 tx=0x7f3e4c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e597fa700 1 -- 172.21.15.77:0/608345339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e4c003930 con 0x7f3e5c107d00 2024-03-16T09:32:35.633 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e597fa700 1 -- 172.21.15.77:0/608345339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3e4c003ad0 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5c19c340 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e597fa700 1 -- 172.21.15.77:0/608345339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e4c0195e0 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.284+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5c19f030 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.285+0000 7f3e597fa700 1 -- 172.21.15.77:0/608345339 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3e4c01a430 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.285+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e5c063900 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.285+0000 7f3e597fa700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3e44038f20 0x7f3e4403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.285+0000 7f3e597fa700 1 -- 172.21.15.77:0/608345339 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3e4c04d2e0 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.285+0000 7f3e5b7fe700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3e44038f20 0x7f3e4403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.286+0000 7f3e5b7fe700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3e44038f20 0x7f3e4403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006fd0 tx=0x7f3e50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.291+0000 7f3e597fa700 1 -- 172.21.15.77:0/608345339 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e4c01a720 con 0x7f3e5c107d00 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:34.485+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3e5c1045e0 con 0x7f3e44038f20 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.014+0000 7f3e597fa700 1 -- 172.21.15.77:0/608345339 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3e5c1045e0 con 0x7f3e44038f20 2024-03-16T09:32:35.634 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3e44038f20 msgr2=0x7f3e4403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3e44038f20 0x7f3e4403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006fd0 tx=0x7f3e50006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 msgr2=0x7f3e5c19bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c19bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006af0 tx=0x7f3e4c011b70 comp rx=0 tx=0).stop 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 shutdown_connections 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f3e44038f20 0x7f3e4403b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 --2- 172.21.15.77:0/608345339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e5c107d00 0x7f3e5c19bb70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 >> 172.21.15.77:0/608345339 conn(0x7f3e5c103340 msgr2=0x7f3e5c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 shutdown_connections 2024-03-16T09:32:35.635 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:35.017+0000 7f3e6250f700 1 -- 172.21.15.77:0/608345339 wait complete. 2024-03-16T09:32:36.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:36 smithi077 ceph-mon[32171]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:32:36.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:36 smithi077 ceph-mon[32171]: Generating ssh key... 2024-03-16T09:32:36.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:36 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:36.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:36 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXcmtWq1c8+9YfKoiZCYOAxrUSr+8YP45Qvhw8eeSZm1c89VL3FvzETx4LcqIz8p56IIX5W4bBxvDZTlq2P1YmFxsbC0pYzlcAXdXepDm73ZGttpcFu3xMvMs+tSb5cJa54YVh6LHjG9Et1RXXblwVdVcPDMKp+CB+HFvykBzrpcKVAmkqB6FhzhOCh30vS+JkM0Jeu5kz8upwJEkq3FGdqqVivbnR3mq4bu9TKTSW55P9C4rjo3q+UmfEMPiQLy+fg7bTLpC3ZWIWhU1kCdKEPl0nPePY3YcC1As9RsGnvm4ofk4GivfwIPnMIcmpllfni+Gq00T8p/Qizva+vH/wJP33XeRxsAqdX7mJWMHWm4qBeJZXoA1iz513Qq3WevNDyKDLj4r45y6PxAPAwddi0u154cVECbI6cONY1FZgCogBg4BhkvULUZN05SL2aLU0wczl8/ycGGQ57v+Yl/3lXDeuB5xd08z5bnhKhmvTQ9dyQoGoUs1w4T0sxnYTf5E= ceph-e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.132+0000 7f0d4e012700 1 Processor -- start 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.132+0000 7f0d4e012700 1 -- start start 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.132+0000 7f0d4e012700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4807c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.132+0000 7f0d4e012700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4807c830 con 0x7f0d4807de00 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.132+0000 7f0d477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4807c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.133+0000 7f0d477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4807c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49702/0 (socket says 172.21.15.77:49702) 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.133+0000 7f0d477fe700 1 -- 172.21.15.77:0/1788796180 learned_addr learned my addr 172.21.15.77:0/1788796180 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.133+0000 7f0d477fe700 1 -- 172.21.15.77:0/1788796180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4807c9b0 con 0x7f0d4807de00 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.133+0000 7f0d477fe700 1 --2- 172.21.15.77:0/1788796180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4807c260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d380098a0 tx=0x7f0d38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309e3be706653b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:36.913 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.133+0000 7f0d467fc700 1 -- 172.21.15.77:0/1788796180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d38004160 con 0x7f0d4807de00 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.133+0000 7f0d467fc700 1 -- 172.21.15.77:0/1788796180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0d38004300 con 0x7f0d4807de00 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.133+0000 7f0d467fc700 1 -- 172.21.15.77:0/1788796180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d38011a80 con 0x7f0d4807de00 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 -- 172.21.15.77:0/1788796180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 msgr2=0x7f0d4807c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 --2- 172.21.15.77:0/1788796180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4807c260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d380098a0 tx=0x7f0d38009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 -- 172.21.15.77:0/1788796180 shutdown_connections 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 --2- 172.21.15.77:0/1788796180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4807c260 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 -- 172.21.15.77:0/1788796180 >> 172.21.15.77:0/1788796180 conn(0x7f0d48103340 msgr2=0x7f0d48105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 -- 172.21.15.77:0/1788796180 shutdown_connections 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 -- 172.21.15.77:0/1788796180 wait complete. 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.134+0000 7f0d4e012700 1 Processor -- start 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d4e012700 1 -- start start 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d4e012700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d4e012700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4810e200 con 0x7f0d4807de00 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49714/0 (socket says 172.21.15.77:49714) 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d477fe700 1 -- 172.21.15.77:0/803355241 learned_addr learned my addr 172.21.15.77:0/803355241 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d477fe700 1 -- 172.21.15.77:0/803355241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4819e250 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d477fe700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4819dd10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d38006af0 tx=0x7f0d38011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d44ff9700 1 -- 172.21.15.77:0/803355241 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d38003930 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4819e4e0 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d44ff9700 1 -- 172.21.15.77:0/803355241 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0d38003ad0 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.135+0000 7f0d44ff9700 1 -- 172.21.15.77:0/803355241 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d380195e0 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.136+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d481a11c0 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.136+0000 7f0d44ff9700 1 -- 172.21.15.77:0/803355241 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f0d38019920 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.136+0000 7f0d44ff9700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0d30038ed0 0x7f0d3003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.136+0000 7f0d44ff9700 1 -- 172.21.15.77:0/803355241 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0d3801aa10 con 0x7f0d4807de00 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.137+0000 7f0d46ffd700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0d30038ed0 0x7f0d3003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:36.915 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.137+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d48063900 con 0x7f0d4807de00 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.137+0000 7f0d46ffd700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0d30038ed0 0x7f0d3003b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c006fd0 tx=0x7f0d3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.142+0000 7f0d44ff9700 1 -- 172.21.15.77:0/803355241 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d3802d960 con 0x7f0d4807de00 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.337+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f0d48104610 con 0x7f0d30038ed0 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.337+0000 7f0d44ff9700 1 -- 172.21.15.77:0/803355241 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f0d48104610 con 0x7f0d30038ed0 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0d30038ed0 msgr2=0x7f0d3003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0d30038ed0 0x7f0d3003b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c006fd0 tx=0x7f0d3c006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 msgr2=0x7f0d4819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4819dd10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d38006af0 tx=0x7f0d38011b70 comp rx=0 tx=0).stop 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 shutdown_connections 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0d30038ed0 0x7f0d3003b390 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 --2- 172.21.15.77:0/803355241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d4807de00 0x7f0d4819dd10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 >> 172.21.15.77:0/803355241 conn(0x7f0d48103340 msgr2=0x7f0d48103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 shutdown_connections 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:36.339+0000 7f0d4e012700 1 -- 172.21.15.77:0/803355241 wait complete. 2024-03-16T09:32:36.916 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-16T09:32:36.917 INFO:teuthology.orchestra.run.smithi077.stdout:Adding key to root@localhost authorized_keys... 2024-03-16T09:32:36.917 INFO:teuthology.orchestra.run.smithi077.stdout:Adding host smithi077... 2024-03-16T09:32:37.240 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:37 smithi077 ceph-mon[32171]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:32:38.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:38 smithi077 ceph-mon[32171]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:32:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:39 smithi077 ceph-mon[32171]: Deploying cephadm binary to smithi077 2024-03-16T09:32:42.444 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Added host 'smithi077' with addr '172.21.15.77' 2024-03-16T09:32:42.444 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.347+0000 7f1fdb808700 1 Processor -- start 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.347+0000 7f1fdb808700 1 -- start start 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fdb808700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd41080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fdb808700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd41086c0 con 0x7f1fd4107cd0 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fd95a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd41080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fd95a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd41080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49728/0 (socket says 172.21.15.77:49728) 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fd95a4700 1 -- 172.21.15.77:0/2125430068 learned_addr learned my addr 172.21.15.77:0/2125430068 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fd95a4700 1 -- 172.21.15.77:0/2125430068 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd4108ed0 con 0x7f1fd4107cd0 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fd95a4700 1 --2- 172.21.15.77:0/2125430068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd41080f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40098a0 tx=0x7f1fc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15df2d8906d09cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fcbfff700 1 -- 172.21.15.77:0/2125430068 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc4004160 con 0x7f1fd4107cd0 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fcbfff700 1 -- 172.21.15.77:0/2125430068 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1fc4004300 con 0x7f1fd4107cd0 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.348+0000 7f1fcbfff700 1 -- 172.21.15.77:0/2125430068 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc4011a80 con 0x7f1fd4107cd0 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.349+0000 7f1fdb808700 1 -- 172.21.15.77:0/2125430068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 msgr2=0x7f1fd41080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.349+0000 7f1fdb808700 1 --2- 172.21.15.77:0/2125430068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd41080f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40098a0 tx=0x7f1fc4009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.349+0000 7f1fdb808700 1 -- 172.21.15.77:0/2125430068 shutdown_connections 2024-03-16T09:32:42.445 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.349+0000 7f1fdb808700 1 --2- 172.21.15.77:0/2125430068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd41080f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.349+0000 7f1fdb808700 1 -- 172.21.15.77:0/2125430068 >> 172.21.15.77:0/2125430068 conn(0x7f1fd41032b0 msgr2=0x7f1fd41056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.349+0000 7f1fdb808700 1 -- 172.21.15.77:0/2125430068 shutdown_connections 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.349+0000 7f1fdb808700 1 -- 172.21.15.77:0/2125430068 wait complete. 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fdb808700 1 Processor -- start 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fdb808700 1 -- start start 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fdb808700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fdb808700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd410c100 con 0x7f1fd4107cd0 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fd95a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fd95a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd419bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49740/0 (socket says 172.21.15.77:49740) 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fd95a4700 1 -- 172.21.15.77:0/3770287380 learned_addr learned my addr 172.21.15.77:0/3770287380 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fd95a4700 1 -- 172.21.15.77:0/3770287380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd419c060 con 0x7f1fd4107cd0 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fd95a4700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd419bb20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4000c00 tx=0x7f1fc4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:42.446 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fca7fc700 1 -- 172.21.15.77:0/3770287380 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc40039f0 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fca7fc700 1 -- 172.21.15.77:0/3770287380 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1fc4003b90 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd419c2f0 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.350+0000 7f1fca7fc700 1 -- 172.21.15.77:0/3770287380 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc401a740 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.351+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd419efe0 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.351+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd4063900 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.351+0000 7f1fca7fc700 1 -- 172.21.15.77:0/3770287380 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f1fc401a9f0 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.352+0000 7f1fca7fc700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1fc0038f70 0x7f1fc003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.352+0000 7f1fca7fc700 1 -- 172.21.15.77:0/3770287380 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1fc404d340 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.352+0000 7f1fd8da3700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1fc0038f70 0x7f1fc003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.352+0000 7f1fd8da3700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1fc0038f70 0x7f1fc003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0006fd0 tx=0x7f1fd0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.356+0000 7f1fca7fc700 1 -- 172.21.15.77:0/3770287380 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1fc401ad90 con 0x7f1fd4107cd0 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:37.552+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}) -- 0x7f1fd4104570 con 0x7f1fc0038f70 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.850+0000 7f1fca7fc700 1 -- 172.21.15.77:0/3770287380 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f1fd4104570 con 0x7f1fc0038f70 2024-03-16T09:32:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1fc0038f70 msgr2=0x7f1fc003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1fc0038f70 0x7f1fc003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0006fd0 tx=0x7f1fd0006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 msgr2=0x7f1fd419bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd419bb20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4000c00 tx=0x7f1fc4011b00 comp rx=0 tx=0).stop 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 shutdown_connections 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f1fc0038f70 0x7f1fc003b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 --2- 172.21.15.77:0/3770287380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fd4107cd0 0x7f1fd419bb20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.852+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 >> 172.21.15.77:0/3770287380 conn(0x7f1fd41032b0 msgr2=0x7f1fd4103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.853+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 shutdown_connections 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:41.853+0000 7f1fdb808700 1 -- 172.21.15.77:0/3770287380 wait complete. 2024-03-16T09:32:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying unmanaged mon service... 2024-03-16T09:32:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:42 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:42 smithi077 ceph-mon[32171]: Added host smithi077 2024-03-16T09:32:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:42 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:44 smithi077 ceph-mon[32171]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:32:44.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:44 smithi077 ceph-mon[32171]: Saving service mon spec with placement count:5 2024-03-16T09:32:44.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:44 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.939+0000 7f318644a700 1 Processor -- start 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.939+0000 7f318644a700 1 -- start start 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.939+0000 7f318644a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180107010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.939+0000 7f318644a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f31801075e0 con 0x7f3180106bf0 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49750/0 (socket says 172.21.15.77:49750) 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317ffff700 1 -- 172.21.15.77:0/1922819881 learned_addr learned my addr 172.21.15.77:0/1922819881 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317ffff700 1 -- 172.21.15.77:0/1922819881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3180107df0 con 0x7f3180106bf0 2024-03-16T09:32:44.768 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317ffff700 1 --2- 172.21.15.77:0/1922819881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180107010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f31700098a0 tx=0x7f3170009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121aea6981bc1338 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317effd700 1 -- 172.21.15.77:0/1922819881 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3170004160 con 0x7f3180106bf0 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317effd700 1 -- 172.21.15.77:0/1922819881 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3170004300 con 0x7f3180106bf0 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.940+0000 7f317effd700 1 -- 172.21.15.77:0/1922819881 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3170011a80 con 0x7f3180106bf0 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.941+0000 7f318644a700 1 -- 172.21.15.77:0/1922819881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 msgr2=0x7f3180107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.941+0000 7f318644a700 1 --2- 172.21.15.77:0/1922819881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180107010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f31700098a0 tx=0x7f3170009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.941+0000 7f318644a700 1 -- 172.21.15.77:0/1922819881 shutdown_connections 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.941+0000 7f318644a700 1 --2- 172.21.15.77:0/1922819881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180107010 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.941+0000 7f318644a700 1 -- 172.21.15.77:0/1922819881 >> 172.21.15.77:0/1922819881 conn(0x7f31801021b0 msgr2=0x7f31801045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.941+0000 7f318644a700 1 -- 172.21.15.77:0/1922819881 shutdown_connections 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.941+0000 7f318644a700 1 -- 172.21.15.77:0/1922819881 wait complete. 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f318644a700 1 Processor -- start 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f318644a700 1 -- start start 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f318644a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f318644a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3180108c80 con 0x7f3180106bf0 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:44.769 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49756/0 (socket says 172.21.15.77:49756) 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317ffff700 1 -- 172.21.15.77:0/943184453 learned_addr learned my addr 172.21.15.77:0/943184453 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317ffff700 1 -- 172.21.15.77:0/943184453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3180199e90 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317ffff700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180199950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3170006af0 tx=0x7f3170011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317d7fa700 1 -- 172.21.15.77:0/943184453 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3170003930 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317d7fa700 1 -- 172.21.15.77:0/943184453 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3170003ad0 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f318019a120 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.942+0000 7f317d7fa700 1 -- 172.21.15.77:0/943184453 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31700195e0 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.943+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318019ce10 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.943+0000 7f317d7fa700 1 -- 172.21.15.77:0/943184453 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3170019890 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.943+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3180063900 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.943+0000 7f317d7fa700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f316803b140 0x7f316803d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.943+0000 7f317d7fa700 1 -- 172.21.15.77:0/943184453 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f317004d5f0 con 0x7f3180106bf0 2024-03-16T09:32:44.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.944+0000 7f317f7fe700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f316803b140 0x7f316803d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.944+0000 7f317f7fe700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f316803b140 0x7f316803d600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3174006fd0 tx=0x7f3174006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:42.949+0000 7f317d7fa700 1 -- 172.21.15.77:0/943184453 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f317000bc40 con 0x7f3180106bf0 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.179+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f3180074b60 con 0x7f316803b140 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.187+0000 7f317d7fa700 1 -- 172.21.15.77:0/943184453 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3180074b60 con 0x7f316803b140 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f316803b140 msgr2=0x7f316803d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f316803b140 0x7f316803d600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3174006fd0 tx=0x7f3174006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 msgr2=0x7f3180199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180199950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3170006af0 tx=0x7f3170011b70 comp rx=0 tx=0).stop 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 shutdown_connections 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f316803b140 0x7f316803d600 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 --2- 172.21.15.77:0/943184453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3180106bf0 0x7f3180199950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 >> 172.21.15.77:0/943184453 conn(0x7f31801021b0 msgr2=0x7f318018ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 shutdown_connections 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:43.189+0000 7f318644a700 1 -- 172.21.15.77:0/943184453 wait complete. 2024-03-16T09:32:44.771 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying unmanaged mgr service... 2024-03-16T09:32:45.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:45 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:45.682 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:45 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:46.720 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:46 smithi077 ceph-mon[32171]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:32:46.720 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:46 smithi077 ceph-mon[32171]: Saving service mgr spec with placement count:2 2024-03-16T09:32:46.720 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:46 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:46.721 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:46 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.230+0000 7f41d0c0b700 1 Processor -- start 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.230+0000 7f41d0c0b700 1 -- start start 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.230+0000 7f41d0c0b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc107010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.230+0000 7f41d0c0b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc1075e0 con 0x7f41cc106bf0 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41ca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41ca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc107010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49772/0 (socket says 172.21.15.77:49772) 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41ca59c700 1 -- 172.21.15.77:0/3745990676 learned_addr learned my addr 172.21.15.77:0/3745990676 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41ca59c700 1 -- 172.21.15.77:0/3745990676 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc107df0 con 0x7f41cc106bf0 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41ca59c700 1 --2- 172.21.15.77:0/3745990676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc107010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41c00098a0 tx=0x7f41c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ce1ab993446f43 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41c959a700 1 -- 172.21.15.77:0/3745990676 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c0004160 con 0x7f41cc106bf0 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41c959a700 1 -- 172.21.15.77:0/3745990676 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41c0004300 con 0x7f41cc106bf0 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.231+0000 7f41c959a700 1 -- 172.21.15.77:0/3745990676 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c0011a50 con 0x7f41cc106bf0 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.232+0000 7f41d0c0b700 1 -- 172.21.15.77:0/3745990676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 msgr2=0x7f41cc107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.232+0000 7f41d0c0b700 1 --2- 172.21.15.77:0/3745990676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc107010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41c00098a0 tx=0x7f41c0009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.232+0000 7f41d0c0b700 1 -- 172.21.15.77:0/3745990676 shutdown_connections 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.232+0000 7f41d0c0b700 1 --2- 172.21.15.77:0/3745990676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc107010 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.232+0000 7f41d0c0b700 1 -- 172.21.15.77:0/3745990676 >> 172.21.15.77:0/3745990676 conn(0x7f41cc1021b0 msgr2=0x7f41cc1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.232+0000 7f41d0c0b700 1 -- 172.21.15.77:0/3745990676 shutdown_connections 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.232+0000 7f41d0c0b700 1 -- 172.21.15.77:0/3745990676 wait complete. 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41d0c0b700 1 Processor -- start 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41d0c0b700 1 -- start start 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41d0c0b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41d0c0b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc10b1a0 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41ca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41ca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc19aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49780/0 (socket says 172.21.15.77:49780) 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41ca59c700 1 -- 172.21.15.77:0/2983570057 learned_addr learned my addr 172.21.15.77:0/2983570057 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41ca59c700 1 -- 172.21.15.77:0/2983570057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc19afc0 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.233+0000 7f41ca59c700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc19aa80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41c0011670 tx=0x7f41c0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.234+0000 7f41bb7fe700 1 -- 172.21.15.77:0/2983570057 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c00038b0 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.234+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41cc19b250 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.234+0000 7f41bb7fe700 1 -- 172.21.15.77:0/2983570057 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41c0003a50 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.234+0000 7f41bb7fe700 1 -- 172.21.15.77:0/2983570057 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c00195c0 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.234+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41cc19df40 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.235+0000 7f41bb7fe700 1 -- 172.21.15.77:0/2983570057 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f41c0019870 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.235+0000 7f41bb7fe700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f41b4038f20 0x7f41b403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.235+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41cc063900 con 0x7f41cc106bf0 2024-03-16T09:32:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.235+0000 7f41bb7fe700 1 -- 172.21.15.77:0/2983570057 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f41c004d3b0 con 0x7f41cc106bf0 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.235+0000 7f41c9d9b700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f41b4038f20 0x7f41b403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.235+0000 7f41c9d9b700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f41b4038f20 0x7f41b403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f41bc006fd0 tx=0x7f41bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.240+0000 7f41bb7fe700 1 -- 172.21.15.77:0/2983570057 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41c001aa70 con 0x7f41cc106bf0 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.483+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f41cc1033f0 con 0x7f41b4038f20 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.490+0000 7f41bb7fe700 1 -- 172.21.15.77:0/2983570057 <== mgr.14120 v2:172.21.15.77:6800/3741578273 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f41cc1033f0 con 0x7f41b4038f20 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f41b4038f20 msgr2=0x7f41b403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f41b4038f20 0x7f41b403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f41bc006fd0 tx=0x7f41bc006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 msgr2=0x7f41cc19aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc19aa80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41c0011670 tx=0x7f41c0011ab0 comp rx=0 tx=0).stop 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 shutdown_connections 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f41b4038f20 0x7f41b403b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 --2- 172.21.15.77:0/2983570057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc106bf0 0x7f41cc19aa80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.492+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 >> 172.21.15.77:0/2983570057 conn(0x7f41cc1021b0 msgr2=0x7f41cc102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:46.732 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.493+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 shutdown_connections 2024-03-16T09:32:46.732 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:45.493+0000 7f41d0c0b700 1 -- 172.21.15.77:0/2983570057 wait complete. 2024-03-16T09:32:47.992 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.168+0000 7f975e837700 1 Processor -- start 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.168+0000 7f975e837700 1 -- start start 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.168+0000 7f975e837700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f97581063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.168+0000 7f975e837700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9758106980 con 0x7f9758105f90 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.168+0000 7f9757fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f97581063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.168+0000 7f9757fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f97581063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55990/0 (socket says 172.21.15.77:55990) 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.168+0000 7f9757fff700 1 -- 172.21.15.77:0/2531280115 learned_addr learned my addr 172.21.15.77:0/2531280115 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.169+0000 7f9757fff700 1 -- 172.21.15.77:0/2531280115 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9758106b00 con 0x7f9758105f90 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.169+0000 7f9757fff700 1 --2- 172.21.15.77:0/2531280115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f97581063b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97480098a0 tx=0x7f9748009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb7bdb2e730d67f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.169+0000 7f9756ffd700 1 -- 172.21.15.77:0/2531280115 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9748004160 con 0x7f9758105f90 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.169+0000 7f9756ffd700 1 -- 172.21.15.77:0/2531280115 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9748004300 con 0x7f9758105f90 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.169+0000 7f9756ffd700 1 -- 172.21.15.77:0/2531280115 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9748011a50 con 0x7f9758105f90 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 -- 172.21.15.77:0/2531280115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 msgr2=0x7f97581063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 --2- 172.21.15.77:0/2531280115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f97581063b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97480098a0 tx=0x7f9748009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 -- 172.21.15.77:0/2531280115 shutdown_connections 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 --2- 172.21.15.77:0/2531280115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f97581063b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 -- 172.21.15.77:0/2531280115 >> 172.21.15.77:0/2531280115 conn(0x7f9758101930 msgr2=0x7f9758103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 -- 172.21.15.77:0/2531280115 shutdown_connections 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 -- 172.21.15.77:0/2531280115 wait complete. 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 Processor -- start 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.170+0000 7f975e837700 1 -- start start 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f975e837700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f9758108890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f975e837700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9758109ea0 con 0x7f9758105f90 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f9757fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f9758108890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f9757fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f9758108890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56002/0 (socket says 172.21.15.77:56002) 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f9757fff700 1 -- 172.21.15.77:0/749998180 learned_addr learned my addr 172.21.15.77:0/749998180 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f9757fff700 1 -- 172.21.15.77:0/749998180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9758108dd0 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f9757fff700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f9758108890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97480117a0 tx=0x7f9748011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f97557fa700 1 -- 172.21.15.77:0/749998180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97480038b0 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f97557fa700 1 -- 172.21.15.77:0/749998180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9748003a50 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9758106fc0 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.171+0000 7f97557fa700 1 -- 172.21.15.77:0/749998180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9748019600 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.172+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9758107500 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.172+0000 7f97557fa700 1 -- 172.21.15.77:0/749998180 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f97480198b0 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.172+0000 7f97557fa700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f9740038f70 0x7f974003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.172+0000 7f97557fa700 1 -- 172.21.15.77:0/749998180 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f974804e2a0 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.173+0000 7f97577fe700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f9740038f70 0x7f974003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.173+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9758063900 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.173+0000 7f97577fe700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f9740038f70 0x7f974003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f974c006fd0 tx=0x7f974c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.178+0000 7f97557fa700 1 -- 172.21.15.77:0/749998180 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97480238c0 con 0x7f9758105f90 2024-03-16T09:32:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.383+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f9758107b10 con 0x7f9758105f90 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.387+0000 7f97557fa700 1 -- 172.21.15.77:0/749998180 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f97480233a0 con 0x7f9758105f90 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f9740038f70 msgr2=0x7f974003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f9740038f70 0x7f974003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f974c006fd0 tx=0x7f974c006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 msgr2=0x7f9758108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f9758108890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97480117a0 tx=0x7f9748011ad0 comp rx=0 tx=0).stop 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 shutdown_connections 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f9740038f70 0x7f974003b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 --2- 172.21.15.77:0/749998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9758105f90 0x7f9758108890 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.390+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 >> 172.21.15.77:0/749998180 conn(0x7f9758101930 msgr2=0x7f9758102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.391+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 shutdown_connections 2024-03-16T09:32:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:47.391+0000 7f975e837700 1 -- 172.21.15.77:0/749998180 wait complete. 2024-03-16T09:32:48.601 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:48 smithi077 ceph-mon[32171]: from='mgr.14120 172.21.15.77:0/417603342' entity='mgr.x' 2024-03-16T09:32:48.601 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:48 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/749998180' entity='client.admin' 2024-03-16T09:32:49.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:49 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1715552402' entity='client.admin' 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.395+0000 7f2c653a3700 1 Processor -- start 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.395+0000 7f2c653a3700 1 -- start start 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.395+0000 7f2c653a3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c60106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.395+0000 7f2c653a3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c601075a0 con 0x7f2c60106bb0 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c60106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c60106fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56010/0 (socket says 172.21.15.77:56010) 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5effd700 1 -- 172.21.15.77:0/818398291 learned_addr learned my addr 172.21.15.77:0/818398291 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5effd700 1 -- 172.21.15.77:0/818398291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c60107db0 con 0x7f2c60106bb0 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5effd700 1 --2- 172.21.15.77:0/818398291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c60106fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c500098a0 tx=0x7f2c50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6a259147eb1f04 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5e7fc700 1 -- 172.21.15.77:0/818398291 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c50004160 con 0x7f2c60106bb0 2024-03-16T09:32:50.080 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5e7fc700 1 -- 172.21.15.77:0/818398291 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c50004300 con 0x7f2c60106bb0 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.396+0000 7f2c5e7fc700 1 -- 172.21.15.77:0/818398291 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c50011a50 con 0x7f2c60106bb0 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 -- 172.21.15.77:0/818398291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 msgr2=0x7f2c60106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 --2- 172.21.15.77:0/818398291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c60106fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c500098a0 tx=0x7f2c50009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 -- 172.21.15.77:0/818398291 shutdown_connections 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 --2- 172.21.15.77:0/818398291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c60106fd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 -- 172.21.15.77:0/818398291 >> 172.21.15.77:0/818398291 conn(0x7f2c601021d0 msgr2=0x7f2c601045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 -- 172.21.15.77:0/818398291 shutdown_connections 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 -- 172.21.15.77:0/818398291 wait complete. 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 Processor -- start 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 -- start start 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c6007ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.397+0000 7f2c653a3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6010b160 con 0x7f2c60106bb0 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c6007ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c6007ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56012/0 (socket says 172.21.15.77:56012) 2024-03-16T09:32:50.081 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5effd700 1 -- 172.21.15.77:0/1715552402 learned_addr learned my addr 172.21.15.77:0/1715552402 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5effd700 1 -- 172.21.15.77:0/1715552402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6019e7c0 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5effd700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c6007ba30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c50011a20 tx=0x7f2c50011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5cff9700 1 -- 172.21.15.77:0/1715552402 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c50003af0 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6019e990 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5cff9700 1 -- 172.21.15.77:0/1715552402 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c5001a8e0 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c5cff9700 1 -- 172.21.15.77:0/1715552402 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c50024860 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.398+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6019edb0 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.399+0000 7f2c5cff9700 1 -- 172.21.15.77:0/1715552402 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2c50022020 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.399+0000 7f2c5cff9700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f2c4c038f20 0x7f2c4c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.399+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c60063900 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.399+0000 7f2c5cff9700 1 -- 172.21.15.77:0/1715552402 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2c5004d280 con 0x7f2c60106bb0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.399+0000 7f2c57fff700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f2c4c038f20 0x7f2c4c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:50.082 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.399+0000 7f2c57fff700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f2c4c038f20 0x7f2c4c03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2c48006fd0 tx=0x7f2c48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.404+0000 7f2c5cff9700 1 -- 172.21.15.77:0/1715552402 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c5000b3e0 con 0x7f2c60106bb0 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.602+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f2c6019f030 con 0x7f2c60106bb0 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.605+0000 7f2c5cff9700 1 -- 172.21.15.77:0/1715552402 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f2c50032030 con 0x7f2c60106bb0 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f2c4c038f20 msgr2=0x7f2c4c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f2c4c038f20 0x7f2c4c03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2c48006fd0 tx=0x7f2c48006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 msgr2=0x7f2c6007ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c6007ba30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c50011a20 tx=0x7f2c50011b00 comp rx=0 tx=0).stop 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 shutdown_connections 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f2c4c038f20 0x7f2c4c03b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 --2- 172.21.15.77:0/1715552402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c60106bb0 0x7f2c6007ba30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 >> 172.21.15.77:0/1715552402 conn(0x7f2c601021d0 msgr2=0x7f2c60102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 shutdown_connections 2024-03-16T09:32:50.083 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:48.608+0000 7f2c653a3700 1 -- 172.21.15.77:0/1715552402 wait complete. 2024-03-16T09:32:50.084 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling the dashboard module... 2024-03-16T09:32:51.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:50 smithi077 ceph-mon[32171]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:32:51.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:50 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/571937523' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-16T09:32:51.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: ignoring --setuser ceph since I am not root 2024-03-16T09:32:51.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: ignoring --setgroup ceph since I am not root 2024-03-16T09:32:51.445 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:51 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:51.349+0000 7fe39f5dd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T09:32:51.445 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:51 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:51.442+0000 7fe39f5dd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T09:32:51.461 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.486+0000 7f0f34333700 1 Processor -- start 2024-03-16T09:32:51.461 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.486+0000 7f0f34333700 1 -- start start 2024-03-16T09:32:51.461 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f34333700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c1080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:51.461 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f34333700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1086a0 con 0x7f0f2c107cb0 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f320cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c1080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f320cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c1080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56028/0 (socket says 172.21.15.77:56028) 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f320cf700 1 -- 172.21.15.77:0/3300337143 learned_addr learned my addr 172.21.15.77:0/3300337143 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f320cf700 1 -- 172.21.15.77:0/3300337143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c108eb0 con 0x7f0f2c107cb0 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f320cf700 1 --2- 172.21.15.77:0/3300337143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c1080d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0f280098a0 tx=0x7f0f28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9f44a5fc2c2e90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f310cd700 1 -- 172.21.15.77:0/3300337143 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f28004160 con 0x7f0f2c107cb0 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f310cd700 1 -- 172.21.15.77:0/3300337143 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f28004300 con 0x7f0f2c107cb0 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.487+0000 7f0f310cd700 1 -- 172.21.15.77:0/3300337143 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f28011a50 con 0x7f0f2c107cb0 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.488+0000 7f0f34333700 1 -- 172.21.15.77:0/3300337143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 msgr2=0x7f0f2c1080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.488+0000 7f0f34333700 1 --2- 172.21.15.77:0/3300337143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c1080d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0f280098a0 tx=0x7f0f28009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.488+0000 7f0f34333700 1 -- 172.21.15.77:0/3300337143 shutdown_connections 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.488+0000 7f0f34333700 1 --2- 172.21.15.77:0/3300337143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c1080d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:51.462 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.488+0000 7f0f34333700 1 -- 172.21.15.77:0/3300337143 >> 172.21.15.77:0/3300337143 conn(0x7f0f2c1032b0 msgr2=0x7f0f2c1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.488+0000 7f0f34333700 1 -- 172.21.15.77:0/3300337143 shutdown_connections 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.488+0000 7f0f34333700 1 -- 172.21.15.77:0/3300337143 wait complete. 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f34333700 1 Processor -- start 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f34333700 1 -- start start 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f34333700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c07daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f34333700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c10c260 con 0x7f0f2c107cb0 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f320cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c07daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f320cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c07daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56044/0 (socket says 172.21.15.77:56044) 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f320cf700 1 -- 172.21.15.77:0/571937523 learned_addr learned my addr 172.21.15.77:0/571937523 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f320cf700 1 -- 172.21.15.77:0/571937523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c07e030 con 0x7f0f2c107cb0 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f320cf700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c07daf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0f28011670 tx=0x7f0f28011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f280038b0 con 0x7f0f2c107cb0 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2c07c220 con 0x7f0f2c107cb0 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.489+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f28003a50 con 0x7f0f2c107cb0 2024-03-16T09:32:51.463 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.490+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f280195c0 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.490+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2c07c760 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.490+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f0f280198e0 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.490+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2c063900 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.491+0000 7f0f237fe700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0f18038ed0 0x7f0f1803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.491+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0f2804d500 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.491+0000 7f0f318ce700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0f18038ed0 0x7f0f1803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.491+0000 7f0f318ce700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0f18038ed0 0x7f0f1803b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006fd0 tx=0x7f0f1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.496+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f28023c90 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.745+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f0f2c07caa0 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.781+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f0f2c07caa0 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.782+0000 7f0f237fe700 1 -- 172.21.15.77:0/571937523 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f0f2801aa70 con 0x7f0f2c107cb0 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0f18038ed0 msgr2=0x7f0f1803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0f18038ed0 0x7f0f1803b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006fd0 tx=0x7f0f1c006e40 comp rx=0 tx=0).stop 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 msgr2=0x7f0f2c07daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:51.464 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c07daf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0f28011670 tx=0x7f0f28011ab0 comp rx=0 tx=0).stop 2024-03-16T09:32:51.465 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 shutdown_connections 2024-03-16T09:32:51.465 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f0f18038ed0 0x7f0f1803b390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:51.465 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 --2- 172.21.15.77:0/571937523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f2c107cb0 0x7f0f2c07daf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:51.465 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 >> 172.21.15.77:0/571937523 conn(0x7f0f2c1032b0 msgr2=0x7f0f2c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:51.465 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 shutdown_connections 2024-03-16T09:32:51.465 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:50.783+0000 7f0f34333700 1 -- 172.21.15.77:0/571937523 wait complete. 2024-03-16T09:32:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:51 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/571937523' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-16T09:32:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:51 smithi077 ceph-mon[32171]: mgrmap e9: x(active, since 21s) 2024-03-16T09:32:52.457 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:52.109+0000 7fe39f5dd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T09:32:52.457 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:52.207+0000 7fe39f5dd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T09:32:52.457 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:52.357+0000 7fe39f5dd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T09:32:52.457 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:52.454+0000 7fe39f5dd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T09:32:52.709 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:52.539+0000 7fe39f5dd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.913+0000 7f80f40c3700 1 Processor -- start 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.913+0000 7f80f40c3700 1 -- start start 2024-03-16T09:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.913+0000 7f80f40c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec10ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.913+0000 7f80f40c3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec07be30 con 0x7f80ec108740 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f1e5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec10ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f1e5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec10ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56066/0 (socket says 172.21.15.77:56066) 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f1e5f700 1 -- 172.21.15.77:0/711610184 learned_addr learned my addr 172.21.15.77:0/711610184 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f1e5f700 1 -- 172.21.15.77:0/711610184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec10b070 con 0x7f80ec108740 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f1e5f700 1 --2- 172.21.15.77:0/711610184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec10ab30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f80e80098a0 tx=0x7f80e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a40b884b523cf37 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f0e5d700 1 -- 172.21.15.77:0/711610184 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e8004160 con 0x7f80ec108740 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f0e5d700 1 -- 172.21.15.77:0/711610184 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80e8004300 con 0x7f80ec108740 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.914+0000 7f80f0e5d700 1 -- 172.21.15.77:0/711610184 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e8011a50 con 0x7f80ec108740 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.915+0000 7f80f40c3700 1 -- 172.21.15.77:0/711610184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 msgr2=0x7f80ec10ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.915+0000 7f80f40c3700 1 --2- 172.21.15.77:0/711610184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec10ab30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f80e80098a0 tx=0x7f80e8009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.915+0000 7f80f40c3700 1 -- 172.21.15.77:0/711610184 shutdown_connections 2024-03-16T09:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.915+0000 7f80f40c3700 1 --2- 172.21.15.77:0/711610184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec10ab30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.915+0000 7f80f40c3700 1 -- 172.21.15.77:0/711610184 >> 172.21.15.77:0/711610184 conn(0x7f80ec1021d0 msgr2=0x7f80ec1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.915+0000 7f80f40c3700 1 -- 172.21.15.77:0/711610184 shutdown_connections 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.915+0000 7f80f40c3700 1 -- 172.21.15.77:0/711610184 wait complete. 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f40c3700 1 Processor -- start 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f40c3700 1 -- start start 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f40c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f40c3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec10c080 con 0x7f80ec108740 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f1e5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f1e5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56076/0 (socket says 172.21.15.77:56076) 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f1e5f700 1 -- 172.21.15.77:0/3870911555 learned_addr learned my addr 172.21.15.77:0/3870911555 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f1e5f700 1 -- 172.21.15.77:0/3870911555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec19c000 con 0x7f80ec108740 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f1e5f700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec19bac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80e8011a20 tx=0x7f80e8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80e2ffd700 1 -- 172.21.15.77:0/3870911555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e80039c0 con 0x7f80ec108740 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80e2ffd700 1 -- 172.21.15.77:0/3870911555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80e8003b60 con 0x7f80ec108740 2024-03-16T09:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ec19c290 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.916+0000 7f80e2ffd700 1 -- 172.21.15.77:0/3870911555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80e801a6b0 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.917+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ec19ef80 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.917+0000 7f80e2ffd700 1 -- 172.21.15.77:0/3870911555 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f80e801a9b0 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.918+0000 7f80e2ffd700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 0x7f80d803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.918+0000 7f80e2ffd700 1 -- 172.21.15.77:0/3870911555 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f80e804d570 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.918+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80ec063900 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.918+0000 7f80f165e700 1 -- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 msgr2=0x7f80d803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3741578273 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.918+0000 7f80f165e700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 0x7f80d803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:51.923+0000 7f80e2ffd700 1 -- 172.21.15.77:0/3870911555 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f80e800b9f0 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.119+0000 7f80f165e700 1 -- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 msgr2=0x7f80d803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3741578273 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.119+0000 7f80f165e700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 0x7f80d803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.182+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f80ec19f150 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.183+0000 7f80e2ffd700 1 -- 172.21.15.77:0/3870911555 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f80e804cec0 con 0x7f80ec108740 2024-03-16T09:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 msgr2=0x7f80d803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 0x7f80d803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 msgr2=0x7f80ec19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec19bac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80e8011a20 tx=0x7f80e8011b00 comp rx=0 tx=0).stop 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 shutdown_connections 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7f80d8038f20 0x7f80d803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 --2- 172.21.15.77:0/3870911555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80ec108740 0x7f80ec19bac0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 >> 172.21.15.77:0/3870911555 conn(0x7f80ec1021d0 msgr2=0x7f80ec102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 shutdown_connections 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:52.185+0000 7f80f40c3700 1 -- 172.21.15.77:0/3870911555 wait complete. 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for the mgr to restart... 2024-03-16T09:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr epoch 9... 2024-03-16T09:32:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3870911555' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T09:32:53.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:52.815+0000 7fe39f5dd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T09:32:53.009 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:52.931+0000 7fe39f5dd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T09:32:53.300 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:53.019+0000 7fe39f5dd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T09:32:53.300 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:53.119+0000 7fe39f5dd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T09:32:53.300 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:53.297+0000 7fe39f5dd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T09:32:54.545 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:54.273+0000 7fe39f5dd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T09:32:54.546 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:54.406+0000 7fe39f5dd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T09:32:54.831 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:54.738+0000 7fe39f5dd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T09:32:55.215 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:54.828+0000 7fe39f5dd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T09:32:55.215 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:55.026+0000 7fe39f5dd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:32:55.215 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:55.212+0000 7fe39f5dd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T09:32:55.759 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:55.500+0000 7fe39f5dd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:32:56.147 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:55.758+0000 7fe39f5dd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T09:32:56.147 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:55.997+0000 7fe39f5dd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T09:32:57.259 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:56.798+0000 7fe39f5dd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T09:32:57.682 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:57.361+0000 7fe39f5dd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T09:32:57.682 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:57.460+0000 7fe39f5dd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T09:32:57.682 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:32:57.679+0000 7fe39f5dd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: Active manager daemon x restarted 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: Activating manager daemon x 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: osdmap e3: 0 total, 0 up, 0 in 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: mgrmap e10: x(active, starting, since 0.00326038s) 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T09:32:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T09:32:58.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T09:32:58.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: Manager daemon x is now available 2024-03-16T09:32:58.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:58.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:32:58.963 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T09:32:58.963 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T09:32:58.963 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:58 smithi077 ceph-mon[32171]: mgrmap e11: x(active, since 1.00569s) 2024-03-16T09:32:59.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T09:32:59.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.135+0000 7fdad7e9b700 1 Processor -- start 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.135+0000 7fdad7e9b700 1 -- start start 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.135+0000 7fdad7e9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad010ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.135+0000 7fdad7e9b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdad007bdf0 con 0x7fdad0108780 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad5c37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad010ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad5c37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad010ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56082/0 (socket says 172.21.15.77:56082) 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad5c37700 1 -- 172.21.15.77:0/569208532 learned_addr learned my addr 172.21.15.77:0/569208532 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad5c37700 1 -- 172.21.15.77:0/569208532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad010b0b0 con 0x7fdad0108780 2024-03-16T09:32:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad5c37700 1 --2- 172.21.15.77:0/569208532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad010ab70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0098a0 tx=0x7fdacc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc52478aca6841aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad4c35700 1 -- 172.21.15.77:0/569208532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdacc004160 con 0x7fdad0108780 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad4c35700 1 -- 172.21.15.77:0/569208532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdacc004300 con 0x7fdad0108780 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.136+0000 7fdad4c35700 1 -- 172.21.15.77:0/569208532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdacc011a50 con 0x7fdad0108780 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.137+0000 7fdad7e9b700 1 -- 172.21.15.77:0/569208532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 msgr2=0x7fdad010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.137+0000 7fdad7e9b700 1 --2- 172.21.15.77:0/569208532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad010ab70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0098a0 tx=0x7fdacc009bb0 comp rx=0 tx=0).stop 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.137+0000 7fdad7e9b700 1 -- 172.21.15.77:0/569208532 shutdown_connections 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.137+0000 7fdad7e9b700 1 --2- 172.21.15.77:0/569208532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad010ab70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.137+0000 7fdad7e9b700 1 -- 172.21.15.77:0/569208532 >> 172.21.15.77:0/569208532 conn(0x7fdad01021d0 msgr2=0x7fdad01045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.137+0000 7fdad7e9b700 1 -- 172.21.15.77:0/569208532 shutdown_connections 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.137+0000 7fdad7e9b700 1 -- 172.21.15.77:0/569208532 wait complete. 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad7e9b700 1 Processor -- start 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad7e9b700 1 -- start start 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad7e9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad019bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:59.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad7e9b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdad010c0b0 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad5c37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad019bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad5c37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad019bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56084/0 (socket says 172.21.15.77:56084) 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad5c37700 1 -- 172.21.15.77:0/3816945926 learned_addr learned my addr 172.21.15.77:0/3816945926 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad5c37700 1 -- 172.21.15.77:0/3816945926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad019c000 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.138+0000 7fdad5c37700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad019bac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdacc011a20 tx=0x7fdacc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.139+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdacc0039c0 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.139+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdacc003b60 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.139+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad019c290 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.139+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdacc01a6b0 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.139+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad019ef80 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.139+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fdacc01a9b0 con 0x7fdad0108780 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.140+0000 7fdac6ffd700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.140+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 --> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdabc03baf0 con 0x7fdabc038f20 2024-03-16T09:32:59.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.140+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdacc04e310 con 0x7fdad0108780 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.140+0000 7fdad5436700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 msgr2=0x7fdabc03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3741578273 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.140+0000 7fdad5436700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.341+0000 7fdad5436700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 msgr2=0x7fdabc03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3741578273 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.341+0000 7fdad5436700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.742+0000 7fdad5436700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 msgr2=0x7fdabc03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3741578273 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:53.742+0000 7fdad5436700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:54.543+0000 7fdad5436700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 msgr2=0x7fdabc03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3741578273 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:54.543+0000 7fdad5436700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:56.144+0000 7fdad5436700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 msgr2=0x7fdabc03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3741578273 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:56.144+0000 7fdad5436700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:57.694+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7fdacc02d910 con 0x7fdad0108780 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:57.694+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 msgr2=0x7fdabc03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:57.694+0000 7fdac6ffd700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:59.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.695+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fdacc04f360 con 0x7fdad0108780 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.695+0000 7fdac6ffd700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdabc03c9a0 0x7fdabc03ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.695+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdacc025e70 con 0x7fdabc03c9a0 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.695+0000 7fdad5436700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdabc03c9a0 0x7fdabc03ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.696+0000 7fdad5436700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdabc03c9a0 0x7fdabc03ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdac0003a10 tx=0x7fdac00092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.697+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fdacc025e70 con 0x7fdabc03c9a0 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.709+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fdad0063900 con 0x7fdabc03c9a0 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.709+0000 7fdac6ffd700 1 -- 172.21.15.77:0/3816945926 <== mgr.14152 v2:172.21.15.77:6800/3114858521 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fdad0063900 con 0x7fdabc03c9a0 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdabc03c9a0 msgr2=0x7fdabc03ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdabc03c9a0 0x7fdabc03ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdac0003a10 tx=0x7fdac00092b0 comp rx=0 tx=0).stop 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 msgr2=0x7fdad019bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad019bac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdacc011a20 tx=0x7fdacc011b00 comp rx=0 tx=0).stop 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 shutdown_connections 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdabc03c9a0 0x7fdabc03ed90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdad0108780 0x7fdad019bac0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 --2- 172.21.15.77:0/3816945926 >> [v2:172.21.15.77:6800/3741578273,v1:172.21.15.77:6801/3741578273] conn(0x7fdabc038f20 0x7fdabc03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:32:59.667 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 >> 172.21.15.77:0/3816945926 conn(0x7fdad01021d0 msgr2=0x7fdad0102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:32:59.668 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 shutdown_connections 2024-03-16T09:32:59.668 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:32:58.710+0000 7fdad7e9b700 1 -- 172.21.15.77:0/3816945926 wait complete. 2024-03-16T09:32:59.668 INFO:teuthology.orchestra.run.smithi077.stdout:mgr epoch 9 is available 2024-03-16T09:32:59.668 INFO:teuthology.orchestra.run.smithi077.stdout:Generating a dashboard self-signed certificate... 2024-03-16T09:32:59.902 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:59 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:57] ENGINE Bus STARTING 2024-03-16T09:32:59.902 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:59 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:58] ENGINE Serving on http://172.21.15.77:8765 2024-03-16T09:32:59.902 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:59 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:58] ENGINE Serving on https://172.21.15.77:7150 2024-03-16T09:32:59.902 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:59 smithi077 ceph-mon[32171]: [16/Mar/2024:09:32:58] ENGINE Bus STARTED 2024-03-16T09:32:59.902 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:59 smithi077 ceph-mon[32171]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T09:32:59.902 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:59 smithi077 ceph-mon[32171]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T09:32:59.902 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:32:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:01.106 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-16T09:33:01.106 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.100+0000 7fc0ebe75700 1 Processor -- start 2024-03-16T09:33:01.106 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.100+0000 7fc0ebe75700 1 -- start start 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.100+0000 7fc0ebe75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e410abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.100+0000 7fc0ebe75700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e407bdd0 con 0x7fc0e41087f0 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.100+0000 7fc0e9c11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e410abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.100+0000 7fc0e9c11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e410abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39064/0 (socket says 172.21.15.77:39064) 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.100+0000 7fc0e9c11700 1 -- 172.21.15.77:0/137542395 learned_addr learned my addr 172.21.15.77:0/137542395 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.101+0000 7fc0e9c11700 1 -- 172.21.15.77:0/137542395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e410b120 con 0x7fc0e41087f0 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.101+0000 7fc0e9c11700 1 --2- 172.21.15.77:0/137542395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e410abe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00098a0 tx=0x7fc0e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f649abcd6439a0ab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.101+0000 7fc0e8c0f700 1 -- 172.21.15.77:0/137542395 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e0004160 con 0x7fc0e41087f0 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.101+0000 7fc0e8c0f700 1 -- 172.21.15.77:0/137542395 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0e0004300 con 0x7fc0e41087f0 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.101+0000 7fc0e8c0f700 1 -- 172.21.15.77:0/137542395 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e0011a50 con 0x7fc0e41087f0 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 -- 172.21.15.77:0/137542395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 msgr2=0x7fc0e410abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 --2- 172.21.15.77:0/137542395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e410abe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00098a0 tx=0x7fc0e0009bb0 comp rx=0 tx=0).stop 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 -- 172.21.15.77:0/137542395 shutdown_connections 2024-03-16T09:33:01.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 --2- 172.21.15.77:0/137542395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e410abe0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 -- 172.21.15.77:0/137542395 >> 172.21.15.77:0/137542395 conn(0x7fc0e4102220 msgr2=0x7fc0e4104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 -- 172.21.15.77:0/137542395 shutdown_connections 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 -- 172.21.15.77:0/137542395 wait complete. 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 Processor -- start 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 -- start start 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0ebe75700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e410c0d0 con 0x7fc0e41087f0 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.102+0000 7fc0e9c11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0e9c11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e419bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39076/0 (socket says 172.21.15.77:39076) 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0e9c11700 1 -- 172.21.15.77:0/382769395 learned_addr learned my addr 172.21.15.77:0/382769395 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0e9c11700 1 -- 172.21.15.77:0/382769395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e419c090 con 0x7fc0e41087f0 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0e9c11700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e419bb50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0011670 tx=0x7fc0e0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e00038b0 con 0x7fc0e41087f0 2024-03-16T09:33:01.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0e0003a50 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e00195c0 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e419c320 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.103+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e419f010 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.104+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fc0e001aa70 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.104+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e4063900 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.104+0000 7fc0daffd700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0d0038e10 0x7fc0d003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.104+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc0e004d360 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.104+0000 7fc0e9410700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0d0038e10 0x7fc0d003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.105+0000 7fc0e9410700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0d0038e10 0x7fc0d003b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006fd0 tx=0x7fc0d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.109+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc0e00212a0 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.311+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fc0e0019c40 con 0x7fc0e41087f0 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.351+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fc0e4103420 con 0x7fc0d0038e10 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.510+0000 7fc0daffd700 1 -- 172.21.15.77:0/382769395 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fc0e4103420 con 0x7fc0d0038e10 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0d0038e10 msgr2=0x7fc0d003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:01.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0d0038e10 0x7fc0d003b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006fd0 tx=0x7fc0d4006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 msgr2=0x7fc0e419bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e419bb50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0011670 tx=0x7fc0e0011ab0 comp rx=0 tx=0).stop 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 shutdown_connections 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0d0038e10 0x7fc0d003b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 --2- 172.21.15.77:0/382769395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e41087f0 0x7fc0e419bb50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 >> 172.21.15.77:0/382769395 conn(0x7fc0e4102220 msgr2=0x7fc0e4102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 shutdown_connections 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:00.513+0000 7fc0ebe75700 1 -- 172.21.15.77:0/382769395 wait complete. 2024-03-16T09:33:01.110 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial admin user... 2024-03-16T09:33:01.351 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:01 smithi077 ceph-mon[32171]: mgrmap e12: x(active, since 2s) 2024-03-16T09:33:01.352 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:01 smithi077 ceph-mon[32171]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:01.352 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:01.352 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:01.352 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:02 smithi077 ceph-mon[32171]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:03.378 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$EEbq7oY74y4XbRny.IA7L.O/fB5sVYzpOLCNziOwjwZzEvZIF3bzO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710581582, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-16T09:33:03.378 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.537+0000 7f3ad1627700 1 Processor -- start 2024-03-16T09:33:03.378 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.537+0000 7f3ad1627700 1 -- start start 2024-03-16T09:33:03.378 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.537+0000 7f3ad1627700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc10ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:03.378 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.537+0000 7f3ad1627700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc0635b0 con 0x7f3acc1087a0 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3acaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc10ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3acaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc10ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39090/0 (socket says 172.21.15.77:39090) 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3acaffd700 1 -- 172.21.15.77:0/1373588312 learned_addr learned my addr 172.21.15.77:0/1373588312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3acaffd700 1 -- 172.21.15.77:0/1373588312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc10b0d0 con 0x7f3acc1087a0 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3acaffd700 1 --2- 172.21.15.77:0/1373588312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc10ab90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40098a0 tx=0x7f3ab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fa1b04ac1fc607 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3ac9ffb700 1 -- 172.21.15.77:0/1373588312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ab4004160 con 0x7f3acc1087a0 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3ac9ffb700 1 -- 172.21.15.77:0/1373588312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ab4004300 con 0x7f3acc1087a0 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.538+0000 7f3ac9ffb700 1 -- 172.21.15.77:0/1373588312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ab4011a50 con 0x7f3acc1087a0 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.539+0000 7f3ad1627700 1 -- 172.21.15.77:0/1373588312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 msgr2=0x7f3acc10ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.539+0000 7f3ad1627700 1 --2- 172.21.15.77:0/1373588312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc10ab90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40098a0 tx=0x7f3ab4009bb0 comp rx=0 tx=0).stop 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.539+0000 7f3ad1627700 1 -- 172.21.15.77:0/1373588312 shutdown_connections 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.539+0000 7f3ad1627700 1 --2- 172.21.15.77:0/1373588312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc10ab90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:03.379 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.539+0000 7f3ad1627700 1 -- 172.21.15.77:0/1373588312 >> 172.21.15.77:0/1373588312 conn(0x7f3acc1021b0 msgr2=0x7f3acc1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.539+0000 7f3ad1627700 1 -- 172.21.15.77:0/1373588312 shutdown_connections 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.539+0000 7f3ad1627700 1 -- 172.21.15.77:0/1373588312 wait complete. 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.540+0000 7f3ad1627700 1 Processor -- start 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.540+0000 7f3ad1627700 1 -- start start 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3ad1627700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3ad1627700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc10c0d0 con 0x7f3acc1087a0 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3acaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3acaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc19bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39098/0 (socket says 172.21.15.77:39098) 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3acaffd700 1 -- 172.21.15.77:0/3689402675 learned_addr learned my addr 172.21.15.77:0/3689402675 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3acaffd700 1 -- 172.21.15.77:0/3689402675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc19c060 con 0x7f3acc1087a0 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3acaffd700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc19bb20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4011a00 tx=0x7f3ab4011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3ac3fff700 1 -- 172.21.15.77:0/3689402675 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ab40039e0 con 0x7f3acc1087a0 2024-03-16T09:33:03.380 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3ac3fff700 1 -- 172.21.15.77:0/3689402675 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ab401a8e0 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3acc19c2f0 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3ac3fff700 1 -- 172.21.15.77:0/3689402675 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ab4024760 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.541+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3acc19efe0 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.542+0000 7f3ac3fff700 1 -- 172.21.15.77:0/3689402675 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f3ab4022020 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.542+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3acc063900 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.543+0000 7f3ac3fff700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ab8038f70 0x7f3ab803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.543+0000 7f3ac3fff700 1 -- 172.21.15.77:0/3689402675 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3ab404d330 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.543+0000 7f3aca7fc700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ab8038f70 0x7f3ab803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.543+0000 7f3aca7fc700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ab8038f70 0x7f3ab803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006fd0 tx=0x7f3abc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.547+0000 7f3ac3fff700 1 -- 172.21.15.77:0/3689402675 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ab400ba30 con 0x7f3acc1087a0 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:01.781+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f3acc19f270 con 0x7f3ab8038f70 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.003+0000 7f3ac3fff700 1 -- 172.21.15.77:0/3689402675 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f3acc19f270 con 0x7f3ab8038f70 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.005+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ab8038f70 msgr2=0x7f3ab803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:03.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.005+0000 7f3ad1627700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ab8038f70 0x7f3ab803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006fd0 tx=0x7f3abc006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.005+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 msgr2=0x7f3acc19bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.005+0000 7f3ad1627700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc19bb20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4011a00 tx=0x7f3ab4011ae0 comp rx=0 tx=0).stop 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.006+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 shutdown_connections 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.006+0000 7f3ad1627700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ab8038f70 0x7f3ab803b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.006+0000 7f3ad1627700 1 --2- 172.21.15.77:0/3689402675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc1087a0 0x7f3acc19bb20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.006+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 >> 172.21.15.77:0/3689402675 conn(0x7f3acc1021b0 msgr2=0x7f3acc102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.006+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 shutdown_connections 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:02.006+0000 7f3ad1627700 1 -- 172.21.15.77:0/3689402675 wait complete. 2024-03-16T09:33:03.382 INFO:teuthology.orchestra.run.smithi077.stdout:Fetching dashboard port number... 2024-03-16T09:33:04.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 8443 2024-03-16T09:33:04.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.787+0000 7fd59f6e8700 1 Processor -- start 2024-03-16T09:33:04.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59f6e8700 1 -- start start 2024-03-16T09:33:04.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59f6e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59810abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:04.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59f6e8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd59807bdb0 con 0x7fd5981087f0 2024-03-16T09:33:04.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59d484700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59810abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:04.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59d484700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59810abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39114/0 (socket says 172.21.15.77:39114) 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59d484700 1 -- 172.21.15.77:0/3769783405 learned_addr learned my addr 172.21.15.77:0/3769783405 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59d484700 1 -- 172.21.15.77:0/3769783405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59810b120 con 0x7fd5981087f0 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd59d484700 1 --2- 172.21.15.77:0/3769783405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59810abe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd5940098a0 tx=0x7fd594009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51ac1a24ad5cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.788+0000 7fd58ffff700 1 -- 172.21.15.77:0/3769783405 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd594004160 con 0x7fd5981087f0 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd58ffff700 1 -- 172.21.15.77:0/3769783405 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd594004300 con 0x7fd5981087f0 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd58ffff700 1 -- 172.21.15.77:0/3769783405 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd594011a50 con 0x7fd5981087f0 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd59f6e8700 1 -- 172.21.15.77:0/3769783405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 msgr2=0x7fd59810abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd59f6e8700 1 --2- 172.21.15.77:0/3769783405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59810abe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd5940098a0 tx=0x7fd594009bb0 comp rx=0 tx=0).stop 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd59f6e8700 1 -- 172.21.15.77:0/3769783405 shutdown_connections 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd59f6e8700 1 --2- 172.21.15.77:0/3769783405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59810abe0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd59f6e8700 1 -- 172.21.15.77:0/3769783405 >> 172.21.15.77:0/3769783405 conn(0x7fd598102200 msgr2=0x7fd598104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd59f6e8700 1 -- 172.21.15.77:0/3769783405 shutdown_connections 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.789+0000 7fd59f6e8700 1 -- 172.21.15.77:0/3769783405 wait complete. 2024-03-16T09:33:04.487 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59f6e8700 1 Processor -- start 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59f6e8700 1 -- start start 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59f6e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59f6e8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd59810c0c0 con 0x7fd5981087f0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59d484700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59d484700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59819bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39124/0 (socket says 172.21.15.77:39124) 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59d484700 1 -- 172.21.15.77:0/61967564 learned_addr learned my addr 172.21.15.77:0/61967564 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59d484700 1 -- 172.21.15.77:0/61967564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59819c010 con 0x7fd5981087f0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.790+0000 7fd59d484700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59819bad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd594011a20 tx=0x7fd594011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.791+0000 7fd58e7fc700 1 -- 172.21.15.77:0/61967564 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5940039c0 con 0x7fd5981087f0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.791+0000 7fd58e7fc700 1 -- 172.21.15.77:0/61967564 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd594003b60 con 0x7fd5981087f0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.791+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59819c2a0 con 0x7fd5981087f0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.791+0000 7fd58e7fc700 1 -- 172.21.15.77:0/61967564 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd59401a6b0 con 0x7fd5981087f0 2024-03-16T09:33:04.488 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.791+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59819ef90 con 0x7fd5981087f0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.792+0000 7fd58e7fc700 1 -- 172.21.15.77:0/61967564 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fd59401a960 con 0x7fd5981087f0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.792+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd598063900 con 0x7fd5981087f0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.792+0000 7fd58e7fc700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd584039020 0x7fd58403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.792+0000 7fd58e7fc700 1 -- 172.21.15.77:0/61967564 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd59404d750 con 0x7fd5981087f0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.792+0000 7fd59cc83700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd584039020 0x7fd58403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.792+0000 7fd59cc83700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd584039020 0x7fd58403b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd588006fd0 tx=0x7fd588006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.797+0000 7fd58e7fc700 1 -- 172.21.15.77:0/61967564 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd59401ad50 con 0x7fd5981087f0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.991+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fd59819f540 con 0x7fd5981087f0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.992+0000 7fd58e7fc700 1 -- 172.21.15.77:0/61967564 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fd594024a70 con 0x7fd5981087f0 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd584039020 msgr2=0x7fd58403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd584039020 0x7fd58403b4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd588006fd0 tx=0x7fd588006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:04.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 msgr2=0x7fd59819bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59819bad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd594011a20 tx=0x7fd594011b00 comp rx=0 tx=0).stop 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 shutdown_connections 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd584039020 0x7fd58403b4e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 --2- 172.21.15.77:0/61967564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5981087f0 0x7fd59819bad0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.994+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 >> 172.21.15.77:0/61967564 conn(0x7fd598102200 msgr2=0x7fd598102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.995+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 shutdown_connections 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:03.995+0000 7fd59f6e8700 1 -- 172.21.15.77:0/61967564 wait complete. 2024-03-16T09:33:04.490 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T09:33:04.491 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T09:33:04.496 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T09:33:04.496 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T09:33:04.496 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T09:33:04.496 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-16T09:33:04.498 INFO:teuthology.orchestra.run.smithi077.stdout:Ceph Dashboard is now available at: 2024-03-16T09:33:04.498 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:04.498 INFO:teuthology.orchestra.run.smithi077.stdout: URL: https://smithi077.front.sepia.ceph.com:8443/ 2024-03-16T09:33:04.498 INFO:teuthology.orchestra.run.smithi077.stdout: User: admin 2024-03-16T09:33:04.498 INFO:teuthology.orchestra.run.smithi077.stdout: Password: pn7dv69lo1 2024-03-16T09:33:04.498 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:04.498 INFO:teuthology.orchestra.run.smithi077.stdout:Saving cluster configuration to /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config directory 2024-03-16T09:33:04.499 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling autotune for osd_memory_target 2024-03-16T09:33:05.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:04 smithi077 ceph-mon[32171]: mgrmap e13: x(active, since 6s) 2024-03-16T09:33:05.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:04 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/61967564' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-16T09:33:05.714 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.901+0000 7f18e28e9700 1 Processor -- start 2024-03-16T09:33:05.714 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.901+0000 7f18e28e9700 1 -- start start 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.901+0000 7f18e28e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc10ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.901+0000 7f18e28e9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc063530 con 0x7f18dc108820 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc10ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc10ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39134/0 (socket says 172.21.15.77:39134) 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18dbfff700 1 -- 172.21.15.77:0/3660886442 learned_addr learned my addr 172.21.15.77:0/3660886442 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18dbfff700 1 -- 172.21.15.77:0/3660886442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18dc106610 con 0x7f18dc108820 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18dbfff700 1 --2- 172.21.15.77:0/3660886442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc10ac10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0098a0 tx=0x7f18cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa3ffd73154b3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18daffd700 1 -- 172.21.15.77:0/3660886442 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18cc004160 con 0x7f18dc108820 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18daffd700 1 -- 172.21.15.77:0/3660886442 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18cc004300 con 0x7f18dc108820 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.902+0000 7f18daffd700 1 -- 172.21.15.77:0/3660886442 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18cc011a50 con 0x7f18dc108820 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 -- 172.21.15.77:0/3660886442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 msgr2=0x7f18dc10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 --2- 172.21.15.77:0/3660886442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc10ac10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0098a0 tx=0x7f18cc009bb0 comp rx=0 tx=0).stop 2024-03-16T09:33:05.715 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 -- 172.21.15.77:0/3660886442 shutdown_connections 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 --2- 172.21.15.77:0/3660886442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc10ac10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 -- 172.21.15.77:0/3660886442 >> 172.21.15.77:0/3660886442 conn(0x7f18dc1021b0 msgr2=0x7f18dc1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 -- 172.21.15.77:0/3660886442 shutdown_connections 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 -- 172.21.15.77:0/3660886442 wait complete. 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 Processor -- start 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.903+0000 7f18e28e9700 1 -- start start 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18e28e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc19dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18e28e9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1076d0 con 0x7f18dc108820 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc19dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc19dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39146/0 (socket says 172.21.15.77:39146) 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18dbfff700 1 -- 172.21.15.77:0/938780363 learned_addr learned my addr 172.21.15.77:0/938780363 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18dbfff700 1 -- 172.21.15.77:0/938780363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18dc19e280 con 0x7f18dc108820 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18dbfff700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc19dd40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f18cc011670 tx=0x7f18cc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:05.716 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18d97fa700 1 -- 172.21.15.77:0/938780363 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18cc0038b0 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18d97fa700 1 -- 172.21.15.77:0/938780363 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18cc003a50 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18dc19e510 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.904+0000 7f18d97fa700 1 -- 172.21.15.77:0/938780363 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18cc0195c0 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.905+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18dc1a1200 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.905+0000 7f18d97fa700 1 -- 172.21.15.77:0/938780363 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f18cc019870 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.905+0000 7f18d97fa700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18c4039020 0x7f18c403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.905+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18dc063900 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.906+0000 7f18d97fa700 1 -- 172.21.15.77:0/938780363 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f18cc04d590 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.906+0000 7f18db7fe700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18c4039020 0x7f18c403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.906+0000 7f18db7fe700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18c4039020 0x7f18c403b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18d0006fd0 tx=0x7f18d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:04.910+0000 7f18d97fa700 1 -- 172.21.15.77:0/938780363 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f18cc006ea0 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.102+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f18dc02de30 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.102+0000 7f18d97fa700 1 -- 172.21.15.77:0/938780363 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f18cc0330c0 con 0x7f18dc108820 2024-03-16T09:33:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.104+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18c4039020 msgr2=0x7f18c403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.104+0000 7f18e28e9700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18c4039020 0x7f18c403b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18d0006fd0 tx=0x7f18d0006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.104+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 msgr2=0x7f18dc19dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.104+0000 7f18e28e9700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc19dd40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f18cc011670 tx=0x7f18cc011ab0 comp rx=0 tx=0).stop 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.105+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 shutdown_connections 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.105+0000 7f18e28e9700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18c4039020 0x7f18c403b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.105+0000 7f18e28e9700 1 --2- 172.21.15.77:0/938780363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18dc108820 0x7f18dc19dd40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.105+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 >> 172.21.15.77:0/938780363 conn(0x7f18dc1021b0 msgr2=0x7f18dc102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.105+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 shutdown_connections 2024-03-16T09:33:05.718 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:05.105+0000 7f18e28e9700 1 -- 172.21.15.77:0/938780363 wait complete. 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad1db2d700 1 Processor -- start 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad1db2d700 1 -- start start 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad1db2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad181069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad1db2d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fad1807bdd0 con 0x7fad181065a0 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad181069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad181069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38138/0 (socket says 172.21.15.77:38138) 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad177fe700 1 -- 172.21.15.77:0/2090466202 learned_addr learned my addr 172.21.15.77:0/2090466202 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.165+0000 7fad177fe700 1 -- 172.21.15.77:0/2090466202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad18106f00 con 0x7fad181065a0 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad177fe700 1 --2- 172.21.15.77:0/2090466202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad181069c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fad080098a0 tx=0x7fad08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=411449d831a829ca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad167fc700 1 -- 172.21.15.77:0/2090466202 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad08004160 con 0x7fad181065a0 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad167fc700 1 -- 172.21.15.77:0/2090466202 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad08004300 con 0x7fad181065a0 2024-03-16T09:33:07.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad167fc700 1 -- 172.21.15.77:0/2090466202 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad08011a50 con 0x7fad181065a0 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad1db2d700 1 -- 172.21.15.77:0/2090466202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 msgr2=0x7fad181069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad1db2d700 1 --2- 172.21.15.77:0/2090466202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad181069c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fad080098a0 tx=0x7fad08009bb0 comp rx=0 tx=0).stop 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad1db2d700 1 -- 172.21.15.77:0/2090466202 shutdown_connections 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad1db2d700 1 --2- 172.21.15.77:0/2090466202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad181069c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.166+0000 7fad1db2d700 1 -- 172.21.15.77:0/2090466202 >> 172.21.15.77:0/2090466202 conn(0x7fad18102220 msgr2=0x7fad18104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.167+0000 7fad1db2d700 1 -- 172.21.15.77:0/2090466202 shutdown_connections 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.167+0000 7fad1db2d700 1 -- 172.21.15.77:0/2090466202 wait complete. 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.167+0000 7fad1db2d700 1 Processor -- start 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.167+0000 7fad1db2d700 1 -- start start 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.167+0000 7fad1db2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad1819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.167+0000 7fad1db2d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fad18107f70 con 0x7fad181065a0 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad1819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad1819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38140/0 (socket says 172.21.15.77:38140) 2024-03-16T09:33:07.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad177fe700 1 -- 172.21.15.77:0/2790847244 learned_addr learned my addr 172.21.15.77:0/2790847244 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad177fe700 1 -- 172.21.15.77:0/2790847244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1819e270 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad177fe700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad1819dd30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fad080117a0 tx=0x7fad08011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad14ff9700 1 -- 172.21.15.77:0/2790847244 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad080038b0 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad14ff9700 1 -- 172.21.15.77:0/2790847244 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad08003a50 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad14ff9700 1 -- 172.21.15.77:0/2790847244 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad08019670 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1819e500 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.168+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad181a11f0 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.169+0000 7fad14ff9700 1 -- 172.21.15.77:0/2790847244 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fad08019970 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.169+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad18063900 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.169+0000 7fad14ff9700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad00038fc0 0x7fad0003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.169+0000 7fad14ff9700 1 -- 172.21.15.77:0/2790847244 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fad0804d8a0 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.169+0000 7fad16ffd700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad00038fc0 0x7fad0003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.170+0000 7fad16ffd700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad00038fc0 0x7fad0003b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fad0c006fd0 tx=0x7fad0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.174+0000 7fad14ff9700 1 -- 172.21.15.77:0/2790847244 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad08019e00 con 0x7fad181065a0 2024-03-16T09:33:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.481+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fad1802ddc0 con 0x7fad181065a0 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.483+0000 7fad14ff9700 1 -- 172.21.15.77:0/2790847244 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fad08023e00 con 0x7fad181065a0 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad00038fc0 msgr2=0x7fad0003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad00038fc0 0x7fad0003b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fad0c006fd0 tx=0x7fad0c006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 msgr2=0x7fad1819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad1819dd30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fad080117a0 tx=0x7fad08011ad0 comp rx=0 tx=0).stop 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 shutdown_connections 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad00038fc0 0x7fad0003b480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 --2- 172.21.15.77:0/2790847244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad181065a0 0x7fad1819dd30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 >> 172.21.15.77:0/2790847244 conn(0x7fad18102220 msgr2=0x7fad18194a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.485+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 shutdown_connections 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T09:33:06.486+0000 7fad1db2d700 1 -- 172.21.15.77:0/2790847244 wait complete. 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-16T09:33:07.066 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout:Or, if you are only running a single cluster on this host: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: ceph telemetry on 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout:For more information see: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:07.067 INFO:teuthology.orchestra.run.smithi077.stdout:Bootstrap complete. 2024-03-16T09:33:07.101 INFO:tasks.cephadm:Fetching config... 2024-03-16T09:33:07.101 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:33:07.101 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-16T09:33:07.116 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-16T09:33:07.116 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:33:07.116 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-16T09:33:07.171 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-16T09:33:07.172 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:33:07.172 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-16T09:33:07.238 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-16T09:33:07.239 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:33:07.239 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-16T09:33:07.294 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-16T09:33:07.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXcmtWq1c8+9YfKoiZCYOAxrUSr+8YP45Qvhw8eeSZm1c89VL3FvzETx4LcqIz8p56IIX5W4bBxvDZTlq2P1YmFxsbC0pYzlcAXdXepDm73ZGttpcFu3xMvMs+tSb5cJa54YVh6LHjG9Et1RXXblwVdVcPDMKp+CB+HFvykBzrpcKVAmkqB6FhzhOCh30vS+JkM0Jeu5kz8upwJEkq3FGdqqVivbnR3mq4bu9TKTSW55P9C4rjo3q+UmfEMPiQLy+fg7bTLpC3ZWIWhU1kCdKEPl0nPePY3YcC1As9RsGnvm4ofk4GivfwIPnMIcmpllfni+Gq00T8p/Qizva+vH/wJP33XeRxsAqdX7mJWMHWm4qBeJZXoA1iz513Qq3WevNDyKDLj4r45y6PxAPAwddi0u154cVECbI6cONY1FZgCogBg4BhkvULUZN05SL2aLU0wczl8/ycGGQ57v+Yl/3lXDeuB5xd08z5bnhKhmvTQ9dyQoGoUs1w4T0sxnYTf5E= ceph-e287e1d6-e377-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T09:33:07.373 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXcmtWq1c8+9YfKoiZCYOAxrUSr+8YP45Qvhw8eeSZm1c89VL3FvzETx4LcqIz8p56IIX5W4bBxvDZTlq2P1YmFxsbC0pYzlcAXdXepDm73ZGttpcFu3xMvMs+tSb5cJa54YVh6LHjG9Et1RXXblwVdVcPDMKp+CB+HFvykBzrpcKVAmkqB6FhzhOCh30vS+JkM0Jeu5kz8upwJEkq3FGdqqVivbnR3mq4bu9TKTSW55P9C4rjo3q+UmfEMPiQLy+fg7bTLpC3ZWIWhU1kCdKEPl0nPePY3YcC1As9RsGnvm4ofk4GivfwIPnMIcmpllfni+Gq00T8p/Qizva+vH/wJP33XeRxsAqdX7mJWMHWm4qBeJZXoA1iz513Qq3WevNDyKDLj4r45y6PxAPAwddi0u154cVECbI6cONY1FZgCogBg4BhkvULUZN05SL2aLU0wczl8/ycGGQ57v+Yl/3lXDeuB5xd08z5bnhKhmvTQ9dyQoGoUs1w4T0sxnYTf5E= ceph-e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:07.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXcmtWq1c8+9YfKoiZCYOAxrUSr+8YP45Qvhw8eeSZm1c89VL3FvzETx4LcqIz8p56IIX5W4bBxvDZTlq2P1YmFxsbC0pYzlcAXdXepDm73ZGttpcFu3xMvMs+tSb5cJa54YVh6LHjG9Et1RXXblwVdVcPDMKp+CB+HFvykBzrpcKVAmkqB6FhzhOCh30vS+JkM0Jeu5kz8upwJEkq3FGdqqVivbnR3mq4bu9TKTSW55P9C4rjo3q+UmfEMPiQLy+fg7bTLpC3ZWIWhU1kCdKEPl0nPePY3YcC1As9RsGnvm4ofk4GivfwIPnMIcmpllfni+Gq00T8p/Qizva+vH/wJP33XeRxsAqdX7mJWMHWm4qBeJZXoA1iz513Qq3WevNDyKDLj4r45y6PxAPAwddi0u154cVECbI6cONY1FZgCogBg4BhkvULUZN05SL2aLU0wczl8/ycGGQ57v+Yl/3lXDeuB5xd08z5bnhKhmvTQ9dyQoGoUs1w4T0sxnYTf5E= ceph-e287e1d6-e377-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T09:33:07.432 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXcmtWq1c8+9YfKoiZCYOAxrUSr+8YP45Qvhw8eeSZm1c89VL3FvzETx4LcqIz8p56IIX5W4bBxvDZTlq2P1YmFxsbC0pYzlcAXdXepDm73ZGttpcFu3xMvMs+tSb5cJa54YVh6LHjG9Et1RXXblwVdVcPDMKp+CB+HFvykBzrpcKVAmkqB6FhzhOCh30vS+JkM0Jeu5kz8upwJEkq3FGdqqVivbnR3mq4bu9TKTSW55P9C4rjo3q+UmfEMPiQLy+fg7bTLpC3ZWIWhU1kCdKEPl0nPePY3YcC1As9RsGnvm4ofk4GivfwIPnMIcmpllfni+Gq00T8p/Qizva+vH/wJP33XeRxsAqdX7mJWMHWm4qBeJZXoA1iz513Qq3WevNDyKDLj4r45y6PxAPAwddi0u154cVECbI6cONY1FZgCogBg4BhkvULUZN05SL2aLU0wczl8/ycGGQ57v+Yl/3lXDeuB5xd08z5bnhKhmvTQ9dyQoGoUs1w4T0sxnYTf5E= ceph-e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:07.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXcmtWq1c8+9YfKoiZCYOAxrUSr+8YP45Qvhw8eeSZm1c89VL3FvzETx4LcqIz8p56IIX5W4bBxvDZTlq2P1YmFxsbC0pYzlcAXdXepDm73ZGttpcFu3xMvMs+tSb5cJa54YVh6LHjG9Et1RXXblwVdVcPDMKp+CB+HFvykBzrpcKVAmkqB6FhzhOCh30vS+JkM0Jeu5kz8upwJEkq3FGdqqVivbnR3mq4bu9TKTSW55P9C4rjo3q+UmfEMPiQLy+fg7bTLpC3ZWIWhU1kCdKEPl0nPePY3YcC1As9RsGnvm4ofk4GivfwIPnMIcmpllfni+Gq00T8p/Qizva+vH/wJP33XeRxsAqdX7mJWMHWm4qBeJZXoA1iz513Qq3WevNDyKDLj4r45y6PxAPAwddi0u154cVECbI6cONY1FZgCogBg4BhkvULUZN05SL2aLU0wczl8/ycGGQ57v+Yl/3lXDeuB5xd08z5bnhKhmvTQ9dyQoGoUs1w4T0sxnYTf5E= ceph-e287e1d6-e377-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T09:33:07.492 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXcmtWq1c8+9YfKoiZCYOAxrUSr+8YP45Qvhw8eeSZm1c89VL3FvzETx4LcqIz8p56IIX5W4bBxvDZTlq2P1YmFxsbC0pYzlcAXdXepDm73ZGttpcFu3xMvMs+tSb5cJa54YVh6LHjG9Et1RXXblwVdVcPDMKp+CB+HFvykBzrpcKVAmkqB6FhzhOCh30vS+JkM0Jeu5kz8upwJEkq3FGdqqVivbnR3mq4bu9TKTSW55P9C4rjo3q+UmfEMPiQLy+fg7bTLpC3ZWIWhU1kCdKEPl0nPePY3YcC1As9RsGnvm4ofk4GivfwIPnMIcmpllfni+Gq00T8p/Qizva+vH/wJP33XeRxsAqdX7mJWMHWm4qBeJZXoA1iz513Qq3WevNDyKDLj4r45y6PxAPAwddi0u154cVECbI6cONY1FZgCogBg4BhkvULUZN05SL2aLU0wczl8/ycGGQ57v+Yl/3lXDeuB5xd08z5bnhKhmvTQ9dyQoGoUs1w4T0sxnYTf5E= ceph-e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:07.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-16T09:33:07.742 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:07.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:07 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2790847244' entity='client.admin' 2024-03-16T09:33:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.997+0000 7fcfff9ab700 1 -- 172.21.15.77:0/1889386659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 msgr2=0x7fcff8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.997+0000 7fcfff9ab700 1 --2- 172.21.15.77:0/1889386659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8102140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b0a0 tx=0x7fcff400b3b0 comp rx=0 tx=0).stop 2024-03-16T09:33:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.997+0000 7fcfff9ab700 1 -- 172.21.15.77:0/1889386659 shutdown_connections 2024-03-16T09:33:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.997+0000 7fcfff9ab700 1 --2- 172.21.15.77:0/1889386659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8102140 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.997+0000 7fcfff9ab700 1 -- 172.21.15.77:0/1889386659 >> 172.21.15.77:0/1889386659 conn(0x7fcff80fd2c0 msgr2=0x7fcff80ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.997+0000 7fcfff9ab700 1 -- 172.21.15.77:0/1889386659 shutdown_connections 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.997+0000 7fcfff9ab700 1 -- 172.21.15.77:0/1889386659 wait complete. 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcfff9ab700 1 Processor -- start 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcfff9ab700 1 -- start start 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcfff9ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcfff9ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcff8107c60 con 0x7fcff8101d20 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcffd747700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcffd747700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38170/0 (socket says 172.21.15.77:38170) 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcffd747700 1 -- 172.21.15.77:0/2008874580 learned_addr learned my addr 172.21.15.77:0/2008874580 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcffd747700 1 -- 172.21.15.77:0/2008874580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff80764d0 con 0x7fcff8101d20 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcffd747700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8075f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcff400d010 tx=0x7fcff4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcfee7fc700 1 -- 172.21.15.77:0/2008874580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcff4017040 con 0x7fcff8101d20 2024-03-16T09:33:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcfee7fc700 1 -- 172.21.15.77:0/2008874580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcff4022070 con 0x7fcff8101d20 2024-03-16T09:33:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.998+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff80746c0 con 0x7fcff8101d20 2024-03-16T09:33:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.999+0000 7fcfee7fc700 1 -- 172.21.15.77:0/2008874580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcff401d830 con 0x7fcff8101d20 2024-03-16T09:33:10.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.999+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff8074c00 con 0x7fcff8101d20 2024-03-16T09:33:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.999+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff81083e0 con 0x7fcff8101d20 2024-03-16T09:33:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:09.999+0000 7fcfee7fc700 1 -- 172.21.15.77:0/2008874580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fcff401dae0 con 0x7fcff8101d20 2024-03-16T09:33:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.000+0000 7fcfee7fc700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe4039020 0x7fcfe403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.000+0000 7fcfee7fc700 1 -- 172.21.15.77:0/2008874580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcff400f050 con 0x7fcff8101d20 2024-03-16T09:33:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.000+0000 7fcffcf46700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe4039020 0x7fcfe403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:10.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.000+0000 7fcffcf46700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe4039020 0x7fcfe403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8006fd0 tx=0x7fcfe8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:10.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.004+0000 7fcfee7fc700 1 -- 172.21.15.77:0/2008874580 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcff4027510 con 0x7fcff8101d20 2024-03-16T09:33:10.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.204+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fcff81906e0 con 0x7fcff8101d20 2024-03-16T09:33:10.207 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:10.207 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:10.207 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:33:10.208 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:10.208 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:10.208 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:10.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.207+0000 7fcfee7fc700 1 -- 172.21.15.77:0/2008874580 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fcff401ddd0 con 0x7fcff8101d20 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe4039020 msgr2=0x7fcfe403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe4039020 0x7fcfe403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8006fd0 tx=0x7fcfe8006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 msgr2=0x7fcff8075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8075f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcff400d010 tx=0x7fcff4003ce0 comp rx=0 tx=0).stop 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 shutdown_connections 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe4039020 0x7fcfe403b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 --2- 172.21.15.77:0/2008874580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff8101d20 0x7fcff8075f90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 >> 172.21.15.77:0/2008874580 conn(0x7fcff80fd2c0 msgr2=0x7fcff80fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 shutdown_connections 2024-03-16T09:33:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:10.209+0000 7fcfff9ab700 1 -- 172.21.15.77:0/2008874580 wait complete. 2024-03-16T09:33:10.972 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-16T09:33:10.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-16T09:33:11.209 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:11 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2008874580' entity='client.admin' 2024-03-16T09:33:11.214 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:11.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:11.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 -- 172.21.15.77:0/2501477525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 msgr2=0x7f605c100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 --2- 172.21.15.77:0/2501477525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c100400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6058009910 tx=0x7f6058009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 -- 172.21.15.77:0/2501477525 shutdown_connections 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 --2- 172.21.15.77:0/2501477525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c100400 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 -- 172.21.15.77:0/2501477525 >> 172.21.15.77:0/2501477525 conn(0x7f605c0fb980 msgr2=0x7f605c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 -- 172.21.15.77:0/2501477525 shutdown_connections 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 -- 172.21.15.77:0/2501477525 wait complete. 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 Processor -- start 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.256+0000 7f6064198700 1 -- start start 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6064198700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6064198700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f605c107c60 con 0x7f605c0fffe0 2024-03-16T09:33:13.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6061f34700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6061f34700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c102840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38186/0 (socket says 172.21.15.77:38186) 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6061f34700 1 -- 172.21.15.77:0/3715251179 learned_addr learned my addr 172.21.15.77:0/3715251179 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6061f34700 1 -- 172.21.15.77:0/3715251179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c102d80 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6061f34700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c102840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6058005800 tx=0x7f6058009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6052ffd700 1 -- 172.21.15.77:0/3715251179 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6058027070 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6052ffd700 1 -- 172.21.15.77:0/3715251179 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f605800be10 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c100f70 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6052ffd700 1 -- 172.21.15.77:0/3715251179 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f605800f460 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.257+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c1014b0 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.258+0000 7f6052ffd700 1 -- 172.21.15.77:0/3715251179 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f605802a070 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.258+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c1083e0 con 0x7f605c0fffe0 2024-03-16T09:33:13.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.258+0000 7f6052ffd700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6048038fc0 0x7f604803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.258+0000 7f6052ffd700 1 -- 172.21.15.77:0/3715251179 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f605801e030 con 0x7f605c0fffe0 2024-03-16T09:33:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.259+0000 7f6061733700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6048038fc0 0x7f604803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:13.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.259+0000 7f6061733700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6048038fc0 0x7f604803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f604c006fd0 tx=0x7f604c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:13.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.264+0000 7f6052ffd700 1 -- 172.21.15.77:0/3715251179 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6058013070 con 0x7f605c0fffe0 2024-03-16T09:33:13.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.468+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f605c190770 con 0x7f6048038fc0 2024-03-16T09:33:13.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.471+0000 7f6052ffd700 1 -- 172.21.15.77:0/3715251179 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f605c190770 con 0x7f6048038fc0 2024-03-16T09:33:13.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.473+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6048038fc0 msgr2=0x7f604803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.473+0000 7f6064198700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6048038fc0 0x7f604803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f604c006fd0 tx=0x7f604c006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.473+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 msgr2=0x7f605c102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.473+0000 7f6064198700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c102840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6058005800 tx=0x7f6058009f60 comp rx=0 tx=0).stop 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.474+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 shutdown_connections 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.474+0000 7f6064198700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6048038fc0 0x7f604803b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.474+0000 7f6064198700 1 --2- 172.21.15.77:0/3715251179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c0fffe0 0x7f605c102840 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.474+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 >> 172.21.15.77:0/3715251179 conn(0x7f605c0fb980 msgr2=0x7f605c0fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.474+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 shutdown_connections 2024-03-16T09:33:13.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:13.474+0000 7f6064198700 1 -- 172.21.15.77:0/3715251179 wait complete. 2024-03-16T09:33:14.226 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi119 2024-03-16T09:33:14.226 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:33:14.227 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.conf 2024-03-16T09:33:14.244 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:33:14.244 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:14.301 INFO:tasks.cephadm:Adding host smithi119 to orchestrator... 2024-03-16T09:33:14.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch host add smithi119 2024-03-16T09:33:14.542 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: Updating smithi077:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:14.740 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:14.741 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:15.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:15 smithi077 ceph-mon[32171]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:15.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:15 smithi077 ceph-mon[32171]: Updating smithi077:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T09:33:15.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.973+0000 7f577f58e700 1 -- 172.21.15.77:0/1036856596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 msgr2=0x7f5778103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.973+0000 7f577f58e700 1 --2- 172.21.15.77:0/1036856596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f5778103240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f576800b0a0 tx=0x7f576800b3b0 comp rx=0 tx=0).stop 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.973+0000 7f577f58e700 1 -- 172.21.15.77:0/1036856596 shutdown_connections 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.973+0000 7f577f58e700 1 --2- 172.21.15.77:0/1036856596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f5778103240 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.973+0000 7f577f58e700 1 -- 172.21.15.77:0/1036856596 >> 172.21.15.77:0/1036856596 conn(0x7f5778077810 msgr2=0x7f57780feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.973+0000 7f577f58e700 1 -- 172.21.15.77:0/1036856596 shutdown_connections 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.973+0000 7f577f58e700 1 -- 172.21.15.77:0/1036856596 wait complete. 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577f58e700 1 Processor -- start 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577f58e700 1 -- start start 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577f58e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f57781976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577f58e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f577810c200 con 0x7f5778102e60 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577d32a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f57781976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:15.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577d32a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f57781976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50174/0 (socket says 172.21.15.77:50174) 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577d32a700 1 -- 172.21.15.77:0/3724965399 learned_addr learned my addr 172.21.15.77:0/3724965399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577d32a700 1 -- 172.21.15.77:0/3724965399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5778197c00 con 0x7f5778102e60 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f577d32a700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f57781976c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f57680076a0 tx=0x7f5768009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f57767fc700 1 -- 172.21.15.77:0/3724965399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5768017040 con 0x7f5778102e60 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.974+0000 7f57767fc700 1 -- 172.21.15.77:0/3724965399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5768004040 con 0x7f5778102e60 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.975+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5778197e90 con 0x7f5778102e60 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.975+0000 7f57767fc700 1 -- 172.21.15.77:0/3724965399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f576801d450 con 0x7f5778102e60 2024-03-16T09:33:15.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.975+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57781982f0 con 0x7f5778102e60 2024-03-16T09:33:15.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.975+0000 7f57767fc700 1 -- 172.21.15.77:0/3724965399 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5768024020 con 0x7f5778102e60 2024-03-16T09:33:15.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.976+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5778069fa0 con 0x7f5778102e60 2024-03-16T09:33:15.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.976+0000 7f57767fc700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5760039020 0x7f576003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:15.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.976+0000 7f57767fc700 1 -- 172.21.15.77:0/3724965399 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5768022020 con 0x7f5778102e60 2024-03-16T09:33:15.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.976+0000 7f577cb29700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5760039020 0x7f576003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:15.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.976+0000 7f577cb29700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5760039020 0x7f576003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f576c006fd0 tx=0x7f576c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:15.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:15.981+0000 7f57767fc700 1 -- 172.21.15.77:0/3724965399 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f576802b020 con 0x7f5778102e60 2024-03-16T09:33:16.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:16.178+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}) -- 0x7f5778198a20 con 0x7f5760039020 2024-03-16T09:33:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:16 smithi077 ceph-mon[32171]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:17 smithi077 ceph-mon[32171]: Deploying cephadm binary to smithi119 2024-03-16T09:33:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:17 smithi077 ceph-mon[32171]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:20.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.503+0000 7f57767fc700 1 -- 172.21.15.77:0/3724965399 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f5778198a20 con 0x7f5760039020 2024-03-16T09:33:20.506 INFO:teuthology.orchestra.run.smithi077.stdout:Added host 'smithi119' with addr '172.21.15.119' 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5760039020 msgr2=0x7f576003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5760039020 0x7f576003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f576c006fd0 tx=0x7f576c006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 msgr2=0x7f57781976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f57781976c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f57680076a0 tx=0x7f5768009800 comp rx=0 tx=0).stop 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 shutdown_connections 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5760039020 0x7f576003b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 --2- 172.21.15.77:0/3724965399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5778102e60 0x7f57781976c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 >> 172.21.15.77:0/3724965399 conn(0x7f5778077810 msgr2=0x7f5778103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 shutdown_connections 2024-03-16T09:33:20.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:20.506+0000 7f577f58e700 1 -- 172.21.15.77:0/3724965399 wait complete. 2024-03-16T09:33:20.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:20 smithi077 ceph-mon[32171]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:20.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:20.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:21.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T09:33:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:21.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:21 smithi077 ceph-mon[32171]: Added host smithi119 2024-03-16T09:33:21.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:21.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 -- 172.21.15.77:0/1914305300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 msgr2=0x7fa8101032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 --2- 172.21.15.77:0/1914305300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa8101032e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009910 tx=0x7fa80c009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 -- 172.21.15.77:0/1914305300 shutdown_connections 2024-03-16T09:33:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 --2- 172.21.15.77:0/1914305300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa8101032e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 -- 172.21.15.77:0/1914305300 >> 172.21.15.77:0/1914305300 conn(0x7fa810077810 msgr2=0x7fa8100feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 -- 172.21.15.77:0/1914305300 shutdown_connections 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 -- 172.21.15.77:0/1914305300 wait complete. 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 Processor -- start 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 -- start start 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa810100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.935+0000 7fa8184de700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa81010c200 con 0x7fa810102f00 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa81627a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa810100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa81627a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa810100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50204/0 (socket says 172.21.15.77:50204) 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa81627a700 1 -- 172.21.15.77:0/3022981021 learned_addr learned my addr 172.21.15.77:0/3022981021 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa81627a700 1 -- 172.21.15.77:0/3022981021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa810102a50 con 0x7fa810102f00 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa81627a700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa810100b60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0056d0 tx=0x7fa80c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa8077fe700 1 -- 172.21.15.77:0/3022981021 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa80c026070 con 0x7fa810102f00 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa8077fe700 1 -- 172.21.15.77:0/3022981021 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa80c004370 con 0x7fa810102f00 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa8077fe700 1 -- 172.21.15.77:0/3022981021 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa80c0039b0 con 0x7fa810102f00 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8101010a0 con 0x7fa810102f00 2024-03-16T09:33:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.936+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa810101560 con 0x7fa810102f00 2024-03-16T09:33:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.937+0000 7fa8077fe700 1 -- 172.21.15.77:0/3022981021 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fa80c029070 con 0x7fa810102f00 2024-03-16T09:33:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.937+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa810069f00 con 0x7fa810102f00 2024-03-16T09:33:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.937+0000 7fa8077fe700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa7fc039020 0x7fa7fc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.937+0000 7fa8077fe700 1 -- 172.21.15.77:0/3022981021 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa80c01e070 con 0x7fa810102f00 2024-03-16T09:33:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.937+0000 7fa815a79700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa7fc039020 0x7fa7fc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:22.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.938+0000 7fa815a79700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa7fc039020 0x7fa7fc03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa800006fd0 tx=0x7fa800006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:22.943+0000 7fa8077fe700 1 -- 172.21.15.77:0/3022981021 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa80c013070 con 0x7fa810102f00 2024-03-16T09:33:23.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:22 smithi077 ceph-mon[32171]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:23.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.148+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa810101790 con 0x7fa7fc039020 2024-03-16T09:33:23.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.149+0000 7fa8077fe700 1 -- 172.21.15.77:0/3022981021 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fa810101790 con 0x7fa7fc039020 2024-03-16T09:33:23.152 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:23.152 INFO:teuthology.orchestra.run.smithi077.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}] 2024-03-16T09:33:23.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa7fc039020 msgr2=0x7fa7fc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa7fc039020 0x7fa7fc03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa800006fd0 tx=0x7fa800006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 msgr2=0x7fa810100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa810100b60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0056d0 tx=0x7fa80c0057b0 comp rx=0 tx=0).stop 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 shutdown_connections 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa7fc039020 0x7fa7fc03b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 --2- 172.21.15.77:0/3022981021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa810102f00 0x7fa810100b60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 >> 172.21.15.77:0/3022981021 conn(0x7fa810077810 msgr2=0x7fa810103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 shutdown_connections 2024-03-16T09:33:23.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:23.152+0000 7fa8184de700 1 -- 172.21.15.77:0/3022981021 wait complete. 2024-03-16T09:33:23.743 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi129 2024-03-16T09:33:23.743 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:33:23.743 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.conf 2024-03-16T09:33:23.761 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:33:23.761 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:23.819 INFO:tasks.cephadm:Adding host smithi129 to orchestrator... 2024-03-16T09:33:23.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch host add smithi129 2024-03-16T09:33:24.058 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:24.426 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:24.426 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:24 smithi077 ceph-mon[32171]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:33:24.426 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:24 smithi077 ceph-mon[32171]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.562+0000 7f3961fa1700 1 -- 172.21.15.77:0/3194631996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 msgr2=0x7f395c107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.562+0000 7f3961fa1700 1 --2- 172.21.15.77:0/3194631996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c107f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3950009910 tx=0x7f3950009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.562+0000 7f3961fa1700 1 -- 172.21.15.77:0/3194631996 shutdown_connections 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.562+0000 7f3961fa1700 1 --2- 172.21.15.77:0/3194631996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c107f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.562+0000 7f3961fa1700 1 -- 172.21.15.77:0/3194631996 >> 172.21.15.77:0/3194631996 conn(0x7f395c0fd600 msgr2=0x7f395c0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.562+0000 7f3961fa1700 1 -- 172.21.15.77:0/3194631996 shutdown_connections 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.562+0000 7f3961fa1700 1 -- 172.21.15.77:0/3194631996 wait complete. 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.563+0000 7f3961fa1700 1 Processor -- start 2024-03-16T09:33:25.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.563+0000 7f3961fa1700 1 -- start start 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.563+0000 7f3961fa1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.563+0000 7f3961fa1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f395c10bfe0 con 0x7f395c107b80 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.563+0000 7f3960f9f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.563+0000 7f3960f9f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40150/0 (socket says 172.21.15.77:40150) 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.563+0000 7f3960f9f700 1 -- 172.21.15.77:0/1309365473 learned_addr learned my addr 172.21.15.77:0/1309365473 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.564+0000 7f3960f9f700 1 -- 172.21.15.77:0/1309365473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c197b20 con 0x7f395c107b80 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.564+0000 7f3960f9f700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c1975e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39500056d0 tx=0x7f39500057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.564+0000 7f3959ffb700 1 -- 172.21.15.77:0/1309365473 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3950026070 con 0x7f395c107b80 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.564+0000 7f3959ffb700 1 -- 172.21.15.77:0/1309365473 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f395000bba0 con 0x7f395c107b80 2024-03-16T09:33:25.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.564+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395c197e10 con 0x7f395c107b80 2024-03-16T09:33:25.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.564+0000 7f3959ffb700 1 -- 172.21.15.77:0/1309365473 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39500037f0 con 0x7f395c107b80 2024-03-16T09:33:25.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.564+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395c19beb0 con 0x7f395c107b80 2024-03-16T09:33:25.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.565+0000 7f3959ffb700 1 -- 172.21.15.77:0/1309365473 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f3950029070 con 0x7f395c107b80 2024-03-16T09:33:25.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.565+0000 7f3959ffb700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f394c039020 0x7f394c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:25.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.565+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395c1031f0 con 0x7f395c107b80 2024-03-16T09:33:25.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.565+0000 7f3959ffb700 1 -- 172.21.15.77:0/1309365473 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f395001e070 con 0x7f395c107b80 2024-03-16T09:33:25.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.565+0000 7f395bfff700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f394c039020 0x7f394c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:25.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.566+0000 7f395bfff700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f394c039020 0x7f394c03b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3948006fd0 tx=0x7f3948006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:25.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.570+0000 7f3959ffb700 1 -- 172.21.15.77:0/1309365473 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3950030070 con 0x7f395c107b80 2024-03-16T09:33:25.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:25.768+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}) -- 0x7f395c198940 con 0x7f394c039020 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: Updating smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: Deploying cephadm binary to smithi129 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: Updating smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:28.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:27 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:29.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:28 smithi077 ceph-mon[32171]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:29.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.953+0000 7f3959ffb700 1 -- 172.21.15.77:0/1309365473 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f395c198940 con 0x7f394c039020 2024-03-16T09:33:29.956 INFO:teuthology.orchestra.run.smithi077.stdout:Added host 'smithi129' with addr '172.21.15.129' 2024-03-16T09:33:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f394c039020 msgr2=0x7f394c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f394c039020 0x7f394c03b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3948006fd0 tx=0x7f3948006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:29.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 msgr2=0x7f395c1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c1975e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39500056d0 tx=0x7f39500057b0 comp rx=0 tx=0).stop 2024-03-16T09:33:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 shutdown_connections 2024-03-16T09:33:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f394c039020 0x7f394c03b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 --2- 172.21.15.77:0/1309365473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f395c107b80 0x7f395c1975e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.955+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 >> 172.21.15.77:0/1309365473 conn(0x7f395c0fd600 msgr2=0x7f395c1096a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.956+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 shutdown_connections 2024-03-16T09:33:29.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:29.956+0000 7f3961fa1700 1 -- 172.21.15.77:0/1309365473 wait complete. 2024-03-16T09:33:30.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T09:33:30.970 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:31.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:30 smithi077 ceph-mon[32171]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:31.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:31.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:31.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:32.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:31 smithi077 ceph-mon[32171]: Added host smithi129 2024-03-16T09:33:32.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:31 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.394+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2991152298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 msgr2=0x7fb5a4100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.394+0000 7fb5aacd4700 1 --2- 172.21.15.77:0/2991152298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4100970 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb598009910 tx=0x7fb598009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.394+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2991152298 shutdown_connections 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.394+0000 7fb5aacd4700 1 --2- 172.21.15.77:0/2991152298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4100970 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.394+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2991152298 >> 172.21.15.77:0/2991152298 conn(0x7fb5a40fbe20 msgr2=0x7fb5a40fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.394+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2991152298 shutdown_connections 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.394+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2991152298 wait complete. 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5aacd4700 1 Processor -- start 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5aacd4700 1 -- start start 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5aacd4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:32.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5aacd4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a410a810 con 0x7fb5a4100590 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5a8a70700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5a8a70700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40158/0 (socket says 172.21.15.77:40158) 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5a8a70700 1 -- 172.21.15.77:0/2762156900 learned_addr learned my addr 172.21.15.77:0/2762156900 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.395+0000 7fb5a8a70700 1 -- 172.21.15.77:0/2762156900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a4198a50 con 0x7fb5a4100590 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.396+0000 7fb5a8a70700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4198510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb5980056d0 tx=0x7fb5980057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.396+0000 7fb5a1ffb700 1 -- 172.21.15.77:0/2762156900 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb598026070 con 0x7fb5a4100590 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.396+0000 7fb5a1ffb700 1 -- 172.21.15.77:0/2762156900 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb59800bba0 con 0x7fb5a4100590 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.396+0000 7fb5a1ffb700 1 -- 172.21.15.77:0/2762156900 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5980037f0 con 0x7fb5a4100590 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.396+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a4198d40 con 0x7fb5a4100590 2024-03-16T09:33:32.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.396+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a4199200 con 0x7fb5a4100590 2024-03-16T09:33:32.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.397+0000 7fb5a1ffb700 1 -- 172.21.15.77:0/2762156900 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb598029070 con 0x7fb5a4100590 2024-03-16T09:33:32.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.397+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a4101a70 con 0x7fb5a4100590 2024-03-16T09:33:32.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.397+0000 7fb5a1ffb700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb594038fc0 0x7fb59403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:32.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.397+0000 7fb5a1ffb700 1 -- 172.21.15.77:0/2762156900 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb59801e070 con 0x7fb5a4100590 2024-03-16T09:33:32.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.397+0000 7fb5a3fff700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb594038fc0 0x7fb59403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:32.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.397+0000 7fb5a3fff700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb594038fc0 0x7fb59403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb590006fd0 tx=0x7fb590006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:32.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.402+0000 7fb5a1ffb700 1 -- 172.21.15.77:0/2762156900 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb598013070 con 0x7fb5a4100590 2024-03-16T09:33:32.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.603+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb5a4102570 con 0x7fb594038fc0 2024-03-16T09:33:32.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.604+0000 7fb5a1ffb700 1 -- 172.21.15.77:0/2762156900 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fb5a4102570 con 0x7fb594038fc0 2024-03-16T09:33:32.607 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:33:32.607 INFO:teuthology.orchestra.run.smithi077.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}] 2024-03-16T09:33:32.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.606+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb594038fc0 msgr2=0x7fb59403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.606+0000 7fb5aacd4700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb594038fc0 0x7fb59403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb590006fd0 tx=0x7fb590006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.606+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 msgr2=0x7fb5a4198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.606+0000 7fb5aacd4700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4198510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb5980056d0 tx=0x7fb5980057b0 comp rx=0 tx=0).stop 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.607+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 shutdown_connections 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.607+0000 7fb5aacd4700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb594038fc0 0x7fb59403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.607+0000 7fb5aacd4700 1 --2- 172.21.15.77:0/2762156900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4100590 0x7fb5a4198510 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.607+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 >> 172.21.15.77:0/2762156900 conn(0x7fb5a40fbe20 msgr2=0x7fb5a4108100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.607+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 shutdown_connections 2024-03-16T09:33:32.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:32.607+0000 7fb5aacd4700 1 -- 172.21.15.77:0/2762156900 wait complete. 2024-03-16T09:33:32.932 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:32 smithi077 ceph-mon[32171]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:32.932 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:33.256 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-16T09:33:33.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-16T09:33:33.488 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:33.910 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:33 smithi077 ceph-mon[32171]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:33:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.964+0000 7f0ec9107700 1 -- 172.21.15.77:0/3237201335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 msgr2=0x7f0ec41022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:34.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.964+0000 7f0ec9107700 1 --2- 172.21.15.77:0/3237201335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec41022f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009910 tx=0x7f0eb4009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:34.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.964+0000 7f0ec9107700 1 -- 172.21.15.77:0/3237201335 shutdown_connections 2024-03-16T09:33:34.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.964+0000 7f0ec9107700 1 --2- 172.21.15.77:0/3237201335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec41022f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:34.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.964+0000 7f0ec9107700 1 -- 172.21.15.77:0/3237201335 >> 172.21.15.77:0/3237201335 conn(0x7f0ec40fd7e0 msgr2=0x7f0ec40ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:34.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.964+0000 7f0ec9107700 1 -- 172.21.15.77:0/3237201335 shutdown_connections 2024-03-16T09:33:34.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.964+0000 7f0ec9107700 1 -- 172.21.15.77:0/3237201335 wait complete. 2024-03-16T09:33:34.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.965+0000 7f0ec9107700 1 Processor -- start 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.965+0000 7f0ec9107700 1 -- start start 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.965+0000 7f0ec9107700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec4197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.965+0000 7f0ec9107700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec410c190 con 0x7f0ec4101f10 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.965+0000 7f0ec2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec4197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.965+0000 7f0ec2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec4197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40180/0 (socket says 172.21.15.77:40180) 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0ec2d9d700 1 -- 172.21.15.77:0/1440232131 learned_addr learned my addr 172.21.15.77:0/1440232131 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0ec2d9d700 1 -- 172.21.15.77:0/1440232131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec4197c60 con 0x7f0ec4101f10 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0ec2d9d700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec4197720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40056d0 tx=0x7f0eb40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:34.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0eabfff700 1 -- 172.21.15.77:0/1440232131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0eb4026070 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0eabfff700 1 -- 172.21.15.77:0/1440232131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eb400bba0 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec4197f50 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0eabfff700 1 -- 172.21.15.77:0/1440232131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0eb40037f0 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.966+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec419bff0 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.967+0000 7f0eabfff700 1 -- 172.21.15.77:0/1440232131 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f0eb4029070 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.967+0000 7f0eabfff700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0eac039020 0x7f0eac03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.967+0000 7f0eabfff700 1 -- 172.21.15.77:0/1440232131 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0eb401e070 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.967+0000 7f0ec259c700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0eac039020 0x7f0eac03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.967+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec41033f0 con 0x7f0ec4101f10 2024-03-16T09:33:34.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.968+0000 7f0ec259c700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0eac039020 0x7f0eac03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8006fd0 tx=0x7f0eb8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:34.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:34.972+0000 7f0eabfff700 1 -- 172.21.15.77:0/1440232131 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0eb4030070 con 0x7f0ec4101f10 2024-03-16T09:33:35.008 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:34 smithi077 ceph-mon[32171]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.171+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f0ec4063900 con 0x7f0ec4101f10 2024-03-16T09:33:35.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.763+0000 7f0eabfff700 1 -- 172.21.15.77:0/1440232131 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f0eb4013070 con 0x7f0ec4101f10 2024-03-16T09:33:35.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.765+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0eac039020 msgr2=0x7f0eac03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:35.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.765+0000 7f0ec9107700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0eac039020 0x7f0eac03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8006fd0 tx=0x7f0eb8006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:35.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.765+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 msgr2=0x7f0ec4197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.765+0000 7f0ec9107700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec4197720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40056d0 tx=0x7f0eb40057b0 comp rx=0 tx=0).stop 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.766+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 shutdown_connections 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.766+0000 7f0ec9107700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0eac039020 0x7f0eac03b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.766+0000 7f0ec9107700 1 --2- 172.21.15.77:0/1440232131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec4101f10 0x7f0ec4197720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.766+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 >> 172.21.15.77:0/1440232131 conn(0x7f0ec40fd7e0 msgr2=0x7f0ec41038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.766+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 shutdown_connections 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:35.766+0000 7f0ec9107700 1 -- 172.21.15.77:0/1440232131 wait complete. 2024-03-16T09:33:35.769 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-03-16T09:33:36.153 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:35 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1440232131' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T09:33:36.491 INFO:tasks.cephadm:Adding mon.a on smithi077 2024-03-16T09:33:36.491 INFO:tasks.cephadm:Adding mon.b on smithi119 2024-03-16T09:33:36.491 INFO:tasks.cephadm:Adding mon.c on smithi129 2024-03-16T09:33:36.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi077:172.21.15.77=a;smithi119:172.21.15.119=b;smithi129:172.21.15.129=c' 2024-03-16T09:33:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:33:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1440232131' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T09:33:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T09:33:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:37.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:33:37.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:37.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:37.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-16T09:33:37.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:37.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:37.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.081+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1332325175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 msgr2=0x7fb1d0103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.081+0000 7fb1d83a5700 1 --2- 172.21.15.129:0/1332325175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d0103250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009910 tx=0x7fb1cc009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.081+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1332325175 shutdown_connections 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.081+0000 7fb1d83a5700 1 --2- 172.21.15.129:0/1332325175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d0103250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.081+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1332325175 >> 172.21.15.129:0/1332325175 conn(0x7fb1d00fc6c0 msgr2=0x7fb1d00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.081+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1332325175 shutdown_connections 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.081+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1332325175 wait complete. 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.082+0000 7fb1d83a5700 1 Processor -- start 2024-03-16T09:33:38.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.082+0000 7fb1d83a5700 1 -- start start 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.082+0000 7fb1d83a5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.082+0000 7fb1d83a5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d010c210 con 0x7fb1d0102e70 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.082+0000 7fb1d6141700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.082+0000 7fb1d6141700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.129:49940/0 (socket says 172.21.15.129:49940) 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.082+0000 7fb1d6141700 1 -- 172.21.15.129:0/1745411331 learned_addr learned my addr 172.21.15.129:0/1745411331 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.083+0000 7fb1d6141700 1 -- 172.21.15.129:0/1745411331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d019d960 con 0x7fb1d0102e70 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.083+0000 7fb1d6141700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d019d420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0056d0 tx=0x7fb1cc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.083+0000 7fb1c77fe700 1 -- 172.21.15.129:0/1745411331 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1cc026070 con 0x7fb1d0102e70 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.083+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d019dbf0 con 0x7fb1d0102e70 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.083+0000 7fb1c77fe700 1 -- 172.21.15.129:0/1745411331 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1cc004370 con 0x7fb1d0102e70 2024-03-16T09:33:38.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.083+0000 7fb1c77fe700 1 -- 172.21.15.129:0/1745411331 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1cc0039b0 con 0x7fb1d0102e70 2024-03-16T09:33:38.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.083+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d01975e0 con 0x7fb1d0102e70 2024-03-16T09:33:38.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.084+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d006aa30 con 0x7fb1d0102e70 2024-03-16T09:33:38.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.084+0000 7fb1c77fe700 1 -- 172.21.15.129:0/1745411331 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb1cc029070 con 0x7fb1d0102e70 2024-03-16T09:33:38.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.084+0000 7fb1c77fe700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb1bc038fc0 0x7fb1bc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:38.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.084+0000 7fb1c77fe700 1 -- 172.21.15.129:0/1745411331 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb1cc01e070 con 0x7fb1d0102e70 2024-03-16T09:33:38.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.084+0000 7fb1d5940700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb1bc038fc0 0x7fb1bc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:38.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.085+0000 7fb1d5940700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb1bc038fc0 0x7fb1bc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:38.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.090+0000 7fb1c77fe700 1 -- 172.21.15.129:0/1745411331 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1cc013070 con 0x7fb1d0102e70 2024-03-16T09:33:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:37 smithi077 ceph-mon[32171]: Updating smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:33:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:37 smithi077 ceph-mon[32171]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:37 smithi077 ceph-mon[32171]: Updating smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T09:33:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.294+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi077:172.21.15.77=a;smithi119:172.21.15.119=b;smithi129:172.21.15.129=c", "target": ["mon-mgr", ""]}) -- 0x7fb1d0198b10 con 0x7fb1bc038fc0 2024-03-16T09:33:38.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.301+0000 7fb1c77fe700 1 -- 172.21.15.129:0/1745411331 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb1d0198b10 con 0x7fb1bc038fc0 2024-03-16T09:33:38.304 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mon update... 2024-03-16T09:33:38.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb1bc038fc0 msgr2=0x7fb1bc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:38.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb1bc038fc0 0x7fb1bc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:38.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 msgr2=0x7fb1d019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:38.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d019d420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0056d0 tx=0x7fb1cc0057b0 comp rx=0 tx=0).stop 2024-03-16T09:33:38.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 shutdown_connections 2024-03-16T09:33:38.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb1bc038fc0 0x7fb1bc03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:38.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 --2- 172.21.15.129:0/1745411331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d0102e70 0x7fb1d019d420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:38.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 >> 172.21.15.129:0/1745411331 conn(0x7fb1d00fc6c0 msgr2=0x7fb1d0105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:38.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 shutdown_connections 2024-03-16T09:33:38.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:38.303+0000 7fb1d83a5700 1 -- 172.21.15.129:0/1745411331 wait complete. 2024-03-16T09:33:38.839 DEBUG:teuthology.orchestra.run.smithi119:mon.b> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.b.service 2024-03-16T09:33:38.842 DEBUG:teuthology.orchestra.run.smithi129:mon.c> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.c.service 2024-03-16T09:33:38.844 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T09:33:38.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T09:33:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:39.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:39.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:33:39.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:33:40.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:39 smithi077 ceph-mon[32171]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi077:172.21.15.77=a;smithi119:172.21.15.119=b;smithi129:172.21.15.129=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:40.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:39 smithi077 ceph-mon[32171]: Saving service mon spec with placement smithi077:172.21.15.77=a;smithi119:172.21.15.119=b;smithi129:172.21.15.129=c;count:3 2024-03-16T09:33:40.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:39 smithi077 ceph-mon[32171]: Deploying daemon mon.c on smithi129 2024-03-16T09:33:41.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:40 smithi077 ceph-mon[32171]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:42.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.192+0000 7feabac18700 1 -- 172.21.15.129:0/864755873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 msgr2=0x7feab41018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:42.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.192+0000 7feabac18700 1 --2- 172.21.15.129:0/864755873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab41018d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009910 tx=0x7feaa8009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.192+0000 7feabac18700 1 -- 172.21.15.129:0/864755873 shutdown_connections 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.192+0000 7feabac18700 1 --2- 172.21.15.129:0/864755873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab41018d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.192+0000 7feabac18700 1 -- 172.21.15.129:0/864755873 >> 172.21.15.129:0/864755873 conn(0x7feab40fbe20 msgr2=0x7feab40fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.192+0000 7feabac18700 1 -- 172.21.15.129:0/864755873 shutdown_connections 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.192+0000 7feabac18700 1 -- 172.21.15.129:0/864755873 wait complete. 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.193+0000 7feabac18700 1 Processor -- start 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.193+0000 7feabac18700 1 -- start start 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.193+0000 7feabac18700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab4196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.193+0000 7feabac18700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feab410b090 con 0x7feab41014f0 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab89b4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab4196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab89b4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab4196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.129:49950/0 (socket says 172.21.15.129:49950) 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab89b4700 1 -- 172.21.15.129:0/2609929327 learned_addr learned my addr 172.21.15.129:0/2609929327 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab89b4700 1 -- 172.21.15.129:0/2609929327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab4196ad0 con 0x7feab41014f0 2024-03-16T09:33:42.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab89b4700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab4196590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feaa80056d0 tx=0x7feaa80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:42.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab1ffb700 1 -- 172.21.15.129:0/2609929327 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feaa8026070 con 0x7feab41014f0 2024-03-16T09:33:42.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab1ffb700 1 -- 172.21.15.129:0/2609929327 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feaa800bba0 con 0x7feab41014f0 2024-03-16T09:33:42.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feabac18700 1 -- 172.21.15.129:0/2609929327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab4196dc0 con 0x7feab41014f0 2024-03-16T09:33:42.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.194+0000 7feab1ffb700 1 -- 172.21.15.129:0/2609929327 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feaa80037f0 con 0x7feab41014f0 2024-03-16T09:33:42.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.195+0000 7feabac18700 1 -- 172.21.15.129:0/2609929327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab419ae60 con 0x7feab41014f0 2024-03-16T09:33:42.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.196+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab41024b0 con 0x7feab41014f0 2024-03-16T09:33:42.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.197+0000 7feab1ffb700 1 -- 172.21.15.129:0/2609929327 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7feaa8029070 con 0x7feab41014f0 2024-03-16T09:33:42.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.198+0000 7feab1ffb700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7feaa4039070 0x7feaa403b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:42.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.198+0000 7feab1ffb700 1 -- 172.21.15.129:0/2609929327 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7feaa801e070 con 0x7feab41014f0 2024-03-16T09:33:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.202+0000 7feab3fff700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7feaa4039070 0x7feaa403b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.203+0000 7feab1ffb700 1 -- 172.21.15.129:0/2609929327 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feaa8013070 con 0x7feab41014f0 2024-03-16T09:33:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.204+0000 7feab3fff700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7feaa4039070 0x7feaa403b530 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feaa0006fd0 tx=0x7feaa0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.496+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7feab4063900 con 0x7feab41014f0 2024-03-16T09:33:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.496+0000 7feab1ffb700 1 -- 172.21.15.129:0/2609929327 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7feaa8030020 con 0x7feab41014f0 2024-03-16T09:33:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:33:42.499 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","modified":"2024-03-16T09:31:58.056995Z","created":"2024-03-16T09:31:58.056995Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-16T09:33:42.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7feaa4039070 msgr2=0x7feaa403b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7feaa4039070 0x7feaa403b530 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feaa0006fd0 tx=0x7feaa0006e40 comp rx=0 tx=0).stop 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 msgr2=0x7feab4196590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab4196590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feaa80056d0 tx=0x7feaa80057b0 comp rx=0 tx=0).stop 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 shutdown_connections 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7feaa4039070 0x7feaa403b530 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 --2- 172.21.15.129:0/2609929327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feab41014f0 0x7feab4196590 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 >> 172.21.15.129:0/2609929327 conn(0x7feab40fbe20 msgr2=0x7feab4102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 shutdown_connections 2024-03-16T09:33:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:42.499+0000 7fea9f7fe700 1 -- 172.21.15.129:0/2609929327 wait complete. 2024-03-16T09:33:42.503 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 1 2024-03-16T09:33:43.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:42 smithi077 ceph-mon[32171]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:43.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:42 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/2609929327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T09:33:43.162 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 systemd[1]: Starting Ceph mon.c for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:33:43.464 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 podman[35822]: 2024-03-16 09:33:43.261017296 +0000 UTC m=+0.018330112 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:33:43.464 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 podman[35822]: 2024-03-16 09:33:43.369794791 +0000 UTC m=+0.127107630 container create a8d3da876c9cdca8c983f6eca9bbd97001dc004242f7761336b8ce035e26c9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:33:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 podman[35822]: 2024-03-16 09:33:43.563596912 +0000 UTC m=+0.320909709 container init a8d3da876c9cdca8c983f6eca9bbd97001dc004242f7761336b8ce035e26c9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T09:33:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 podman[35822]: 2024-03-16 09:33:43.566434459 +0000 UTC m=+0.323747257 container start a8d3da876c9cdca8c983f6eca9bbd97001dc004242f7761336b8ce035e26c9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pidfile_write: ignore empty --pid-file 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: load: jerasure load: lrc 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: RocksDB version: 7.9.2 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Git sha 0 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: DB SUMMARY 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: DB Session ID: 6LJAY8MXXTCV6RD3S18N 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: CURRENT file: CURRENT 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-16T09:33:44.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.error_if_exists: 0 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.create_if_missing: 0 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.paranoid_checks: 1 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.env: 0x55bc4ad37c60 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.info_log: 0x55bc4cfc7ac0 2024-03-16T09:33:44.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T09:33:44.054 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.statistics: (nil) 2024-03-16T09:33:44.054 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.use_fsync: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_log_file_size: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.allow_fallocate: 1 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.use_direct_reads: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.db_log_dir: 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.wal_dir: 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T09:33:44.055 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.write_buffer_manager: 0x55bc4cf9a1e0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T09:33:44.056 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.unordered_write: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.row_cache: None 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.wal_filter: None 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.two_write_queues: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.wal_compression: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.atomic_flush: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T09:33:44.057 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.log_readahead_size: 0 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_background_jobs: 2 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_background_compactions: -1 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_subcompactions: 1 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T09:33:44.058 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_open_files: -1 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_background_flushes: -1 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Compression algorithms supported: 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kZSTD supported: 0 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kXpressCompression supported: 0 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T09:33:44.059 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kZlibCompression supported: 1 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kSnappyCompression supported: 1 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kLZ4Compression supported: 1 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: kBZip2Compression supported: 0 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.merge_operator: 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_filter: None 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc4cfc7bc0) 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-03-16T09:33:44.060 INFO:journalctl@ceph.mon.c.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: index_type: 0 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: data_block_index_type: 0 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: index_shortening: 1 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: checksum: 4 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: no_block_cache: 0 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache: 0x55bc4cfcd090 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache_options: 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: capacity : 536870912 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: num_shard_bits : 4 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: strict_capacity_limit : 0 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-03-16T09:33:44.061 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache_compressed: (nil) 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: persistent_cache: (nil) 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_size: 4096 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_size_deviation: 10 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_restart_interval: 16 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: index_block_restart_interval: 1 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: metadata_block_size: 4096 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: partition_filters: 0 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: use_delta_encoding: 1 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: filter_policy: bloomfilter 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: whole_key_filtering: 1 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: verify_compression: 0 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: format_version: 5 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: enable_index_compression: 1 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_align: 0 2024-03-16T09:33:44.062 INFO:journalctl@ceph.mon.c.smithi129.stdout: max_auto_readahead_size: 262144 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout: prepopulate_block_cache: 0 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout: initial_auto_readahead_size: 8192 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression: NoCompression 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.num_levels: 7 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T09:33:44.063 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T09:33:44.064 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T09:33:44.065 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.table_properties_collectors: 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.inplace_update_support: 0 2024-03-16T09:33:44.066 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.bloom_locality: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.max_successive_merges: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.ttl: 2592000 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.enable_blob_files: false 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.min_blob_size: 0 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T09:33:44.067 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3e42c5e8-cf2c-48b6-af64-c146d2f4c596 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710581623607899, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710581623608332, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710581623, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3e42c5e8-cf2c-48b6-af64-c146d2f4c596", "db_session_id": "6LJAY8MXXTCV6RD3S18N", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710581623608430, "job": 1, "event": "recovery_finished"} 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T09:33:44.068 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bc4d0c0000 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: DB pointer 0x55bc4d0aa000 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** DB Stats ** 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T09:33:44.069 INFO:journalctl@ceph.mon.c.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T09:33:44.070 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: Block cache BinnedLRUCache@0x55bc4cfcd090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: using public_addr v2:172.21.15.129:0/0 -> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: starting mon.c rank -1 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(???) e0 preinit fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).mds e1 new map 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: e1 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T09:33:44.071 INFO:journalctl@ceph.mon.c.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout: legacy client fscid: -1 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout: No filesystems configured 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.072 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:33:44.073 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Deploying cephadm binary to smithi129 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.074 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Added host smithi129 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1440232131' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1440232131' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:44.075 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Updating smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi077:172.21.15.77=a;smithi119:172.21.15.119=b;smithi129:172.21.15.129=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Saving service mon spec with placement smithi077:172.21.15.77=a;smithi119:172.21.15.119=b;smithi129:172.21.15.129=c;count:3 2024-03-16T09:33:44.076 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: Deploying daemon mon.c on smithi129 2024-03-16T09:33:44.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:44.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/2609929327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T09:33:44.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 ceph-mon[35841]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T09:33:44.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 bash[35822]: a8d3da876c9cdca8c983f6eca9bbd97001dc004242f7761336b8ce035e26c9c5 2024-03-16T09:33:44.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:43 smithi129 systemd[1]: Started Ceph mon.c for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:33:44.197 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T09:33:44.197 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T09:33:44.433 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:33:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.746+0000 7efd035e6700 1 -- 172.21.15.129:0/1758948531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efce80035c0 msgr2=0x7efce8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.746+0000 7efd035e6700 1 --2- 172.21.15.129:0/1758948531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efce80035c0 0x7efce8005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efcfc101ce0 tx=0x7efcf4009e80 comp rx=0 tx=0).stop 2024-03-16T09:33:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.746+0000 7efd035e6700 1 -- 172.21.15.129:0/1758948531 shutdown_connections 2024-03-16T09:33:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.746+0000 7efd035e6700 1 --2- 172.21.15.129:0/1758948531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efce80035c0 0x7efce8005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.746+0000 7efd035e6700 1 -- 172.21.15.129:0/1758948531 >> 172.21.15.129:0/1758948531 conn(0x7efcfc0fd8a0 msgr2=0x7efcfc0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.746+0000 7efd035e6700 1 -- 172.21.15.129:0/1758948531 shutdown_connections 2024-03-16T09:33:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.746+0000 7efd035e6700 1 -- 172.21.15.129:0/1758948531 wait complete. 2024-03-16T09:33:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.747+0000 7efd035e6700 1 Processor -- start 2024-03-16T09:33:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.747+0000 7efd035e6700 1 -- start start 2024-03-16T09:33:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.747+0000 7efd035e6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 0x7efcfc19d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.747+0000 7efd035e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc19db10 0x7efcfc1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.747+0000 7efd035e6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc109cf0 con 0x7efcfc19db10 2024-03-16T09:33:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.747+0000 7efd035e6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc109b70 con 0x7efce80035c0 2024-03-16T09:33:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.747+0000 7efd01382700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 0x7efcfc19d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd01382700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 0x7efcfc19d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52330/0 (socket says 172.21.15.129:52330) 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd01382700 1 -- 172.21.15.129:0/2628346962 learned_addr learned my addr 172.21.15.129:0/2628346962 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd01382700 1 -- 172.21.15.129:0/2628346962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 msgr2=0x7efcfc19d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd00b81700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc19db10 0x7efcfc1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd01382700 1 -- 172.21.15.129:0/2628346962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 msgr2=0x7efcfc19d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd01382700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 0x7efcfc19d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd01382700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 0x7efcfc19d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd00b81700 1 -- 172.21.15.129:0/2628346962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 msgr2=0x7efcfc19d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd00b81700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 0x7efcfc19d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd00b81700 1 -- 172.21.15.129:0/2628346962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcfc19e090 con 0x7efcfc19db10 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd00b81700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc19db10 0x7efcfc1976f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efcec00b4d0 tx=0x7efcec00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efcf27fc700 1 -- 172.21.15.129:0/2628346962 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efcec011630 con 0x7efcfc19db10 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efcf27fc700 1 -- 172.21.15.129:0/2628346962 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcec011c80 con 0x7efcfc19db10 2024-03-16T09:33:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efcf27fc700 1 -- 172.21.15.129:0/2628346962 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efcec010da0 con 0x7efcfc19db10 2024-03-16T09:33:48.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.748+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcfc197c90 con 0x7efcfc19db10 2024-03-16T09:33:48.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.749+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcfc198220 con 0x7efcfc19db10 2024-03-16T09:33:48.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.749+0000 7efcf27fc700 1 -- 172.21.15.129:0/2628346962 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7efcec018020 con 0x7efcfc19db10 2024-03-16T09:33:48.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.750+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcfc1034f0 con 0x7efcfc19db10 2024-03-16T09:33:48.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.750+0000 7efcf27fc700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efce0038dd0 0x7efce003b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:48.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.750+0000 7efcf27fc700 1 -- 172.21.15.129:0/2628346962 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efcec04ba80 con 0x7efcfc19db10 2024-03-16T09:33:48.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.750+0000 7efd01382700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efce0038dd0 0x7efce003b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:48.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.750+0000 7efd01382700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efce0038dd0 0x7efce003b290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efcf4007590 tx=0x7efcf40074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:48.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:48.755+0000 7efcf27fc700 1 -- 172.21.15.129:0/2628346962 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efcec01f020 con 0x7efcfc19db10 2024-03-16T09:33:49.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: mon.a calling monitor election 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: mon.c calling monitor election 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: monmap epoch 2 2024-03-16T09:33:49.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: last_changed 2024-03-16T09:33:43.632515+0000 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: created 2024-03-16T09:31:58.056995+0000 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: min_mon_release 19 (squid) 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: election_strategy: 1 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.c 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: fsmap 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: mgrmap e13: x(active, since 50s) 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: overall HEALTH_OK 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:33:49.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:49.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.031+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7efcfc198590 con 0x7efcfc19db10 2024-03-16T09:33:49.033 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: mon.a calling monitor election 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: mon.c calling monitor election 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: monmap epoch 2 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:49.034 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: last_changed 2024-03-16T09:33:43.632515+0000 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: created 2024-03-16T09:31:58.056995+0000 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: min_mon_release 19 (squid) 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: election_strategy: 1 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.c 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: fsmap 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: mgrmap e13: x(active, since 50s) 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: overall HEALTH_OK 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:33:49.035 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.031+0000 7efcf27fc700 1 -- 172.21.15.129:0/2628346962 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7efcec01ed10 con 0x7efcfc19db10 2024-03-16T09:33:49.036 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:33:49.036 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","modified":"2024-03-16T09:33:43.632515Z","created":"2024-03-16T09:31:58.056995Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efce0038dd0 msgr2=0x7efce003b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efce0038dd0 0x7efce003b290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efcf4007590 tx=0x7efcf40074e0 comp rx=0 tx=0).stop 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc19db10 msgr2=0x7efcfc1976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc19db10 0x7efcfc1976f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efcec00b4d0 tx=0x7efcec00b890 comp rx=0 tx=0).stop 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 shutdown_connections 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efce0038dd0 0x7efce003b290 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80035c0 0x7efcfc19d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 --2- 172.21.15.129:0/2628346962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcfc19db10 0x7efcfc1976f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 >> 172.21.15.129:0/2628346962 conn(0x7efcfc0fd8a0 msgr2=0x7efcfc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 shutdown_connections 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:49.033+0000 7efd035e6700 1 -- 172.21.15.129:0/2628346962 wait complete. 2024-03-16T09:33:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 2 2024-03-16T09:33:49.857 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:49 smithi129 ceph-mon[35841]: Deploying daemon mon.b on smithi119 2024-03-16T09:33:49.857 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:49 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/2628346962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T09:33:49.857 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:49 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:50.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:49 smithi077 ceph-mon[32171]: Deploying daemon mon.b on smithi119 2024-03-16T09:33:50.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:49 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/2628346962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T09:33:50.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:49 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:50.596 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T09:33:50.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T09:33:50.828 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:33:51.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:50 smithi077 ceph-mon[32171]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:51.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:50 smithi129 ceph-mon[35841]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:52.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.140+0000 7fe458bca700 1 -- 172.21.15.129:0/2743313020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454100ef0 msgr2=0x7fe4541012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:52.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.140+0000 7fe458bca700 1 --2- 172.21.15.129:0/2743313020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454100ef0 0x7fe4541012d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00b0a0 tx=0x7fe44c00b3b0 comp rx=0 tx=0).stop 2024-03-16T09:33:52.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.140+0000 7fe458bca700 1 -- 172.21.15.129:0/2743313020 shutdown_connections 2024-03-16T09:33:52.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.140+0000 7fe458bca700 1 --2- 172.21.15.129:0/2743313020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454100ef0 0x7fe4541012d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:52.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.140+0000 7fe458bca700 1 -- 172.21.15.129:0/2743313020 >> 172.21.15.129:0/2743313020 conn(0x7fe4540fc780 msgr2=0x7fe4540feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:52.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.140+0000 7fe458bca700 1 -- 172.21.15.129:0/2743313020 shutdown_connections 2024-03-16T09:33:52.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.140+0000 7fe458bca700 1 -- 172.21.15.129:0/2743313020 wait complete. 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.141+0000 7fe458bca700 1 Processor -- start 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.141+0000 7fe458bca700 1 -- start start 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.141+0000 7fe458bca700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 0x7fe45419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.141+0000 7fe458bca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe45419db40 0x7fe4541976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.141+0000 7fe458bca700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe45410b170 con 0x7fe45419db40 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.141+0000 7fe458bca700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe45410aff0 con 0x7fe454100ef0 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe45259c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 0x7fe45419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe451d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe45419db40 0x7fe4541976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe45259c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 0x7fe45419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52334/0 (socket says 172.21.15.129:52334) 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe45259c700 1 -- 172.21.15.129:0/2297146899 learned_addr learned my addr 172.21.15.129:0/2297146899 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe45259c700 1 -- 172.21.15.129:0/2297146899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 msgr2=0x7fe45419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T09:33:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe45259c700 1 -- 172.21.15.129:0/2297146899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 msgr2=0x7fe45419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe45259c700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 0x7fe45419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe45259c700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 0x7fe45419d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe451d9b700 1 -- 172.21.15.129:0/2297146899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 msgr2=0x7fe45419d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe451d9b700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 0x7fe45419d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe451d9b700 1 -- 172.21.15.129:0/2297146899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45419e0c0 con 0x7fe45419db40 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe451d9b700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe45419db40 0x7fe4541976d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe44800b4d0 tx=0x7fe44800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe448011630 con 0x7fe45419db40 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe454197c70 con 0x7fe45419db40 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.142+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe448011c80 con 0x7fe45419db40 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.143+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe448010da0 con 0x7fe45419db40 2024-03-16T09:33:52.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.143+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe454198200 con 0x7fe45419db40 2024-03-16T09:33:52.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.144+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4541023d0 con 0x7fe45419db40 2024-03-16T09:33:52.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.144+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fe448018020 con 0x7fe45419db40 2024-03-16T09:33:52.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.144+0000 7fe4437fe700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe43c038d60 0x7fe43c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:52.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.144+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe44804ba40 con 0x7fe45419db40 2024-03-16T09:33:52.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.144+0000 7fe45259c700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe43c038d60 0x7fe43c03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:52.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.144+0000 7fe45259c700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe43c038d60 0x7fe43c03b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe44c015040 tx=0x7fe44c0096e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.149+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe44801f020 con 0x7fe45419db40 2024-03-16T09:33:52.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.370+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4480177d0 con 0x7fe45419db40 2024-03-16T09:33:52.390 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:52 smithi119 ceph-mon[35632]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T09:33:52.391 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:52 smithi119 bash[35614]: 84d0a0b3dd423879946f1a0fba0ed0ffaa397281a529e82e34b8efe3f15ff706 2024-03-16T09:33:52.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:52.423+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe454198570 con 0x7fe45419db40 2024-03-16T09:33:52.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:52 smithi119 systemd[1]: Started Ceph mon.b for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:33:57.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.416+0000 7fe4437fe700 1 -- 172.21.15.129:0/2297146899 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7fe44804be20 con 0x7fe45419db40 2024-03-16T09:33:57.419 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-16T09:33:57.419 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","modified":"2024-03-16T09:33:52.370140Z","created":"2024-03-16T09:31:58.056995Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe43c038d60 msgr2=0x7fe43c03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe43c038d60 0x7fe43c03b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe44c015040 tx=0x7fe44c0096e0 comp rx=0 tx=0).stop 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe45419db40 msgr2=0x7fe4541976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe45419db40 0x7fe4541976d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe44800b4d0 tx=0x7fe44800b890 comp rx=0 tx=0).stop 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 shutdown_connections 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe43c038d60 0x7fe43c03b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe454100ef0 0x7fe45419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 --2- 172.21.15.129:0/2297146899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe45419db40 0x7fe4541976d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.419+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 >> 172.21.15.129:0/2297146899 conn(0x7fe4540fc780 msgr2=0x7fe454069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.420+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 shutdown_connections 2024-03-16T09:33:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:33:57.420+0000 7fe458bca700 1 -- 172.21.15.129:0/2297146899 wait complete. 2024-03-16T09:33:57.424 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 3 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: mon.a calling monitor election 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: mon.c calling monitor election 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/2297146899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: mon.b calling monitor election 2024-03-16T09:33:57.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: monmap epoch 3 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: last_changed 2024-03-16T09:33:52.370140+0000 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: created 2024-03-16T09:31:58.056995+0000 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: min_mon_release 19 (squid) 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: election_strategy: 1 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.c 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: fsmap 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: mgrmap e13: x(active, since 59s) 2024-03-16T09:33:57.748 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: overall HEALTH_OK 2024-03-16T09:33:57.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:33:57.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: mon.a calling monitor election 2024-03-16T09:33:57.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: mon.c calling monitor election 2024-03-16T09:33:57.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/2297146899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: mon.b calling monitor election 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: monmap epoch 3 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: last_changed 2024-03-16T09:33:52.370140+0000 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: created 2024-03-16T09:31:58.056995+0000 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: min_mon_release 19 (squid) 2024-03-16T09:33:57.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: election_strategy: 1 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.c 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: fsmap 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: mgrmap e13: x(active, since 59s) 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: overall HEALTH_OK 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: mon.a calling monitor election 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: mon.c calling monitor election 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/2297146899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:57.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: mon.b calling monitor election 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: monmap epoch 3 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: last_changed 2024-03-16T09:33:52.370140+0000 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: created 2024-03-16T09:31:58.056995+0000 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: min_mon_release 19 (squid) 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: election_strategy: 1 2024-03-16T09:33:57.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.c 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: fsmap 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: mgrmap e13: x(active, since 59s) 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: overall HEALTH_OK 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:57.942 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:58.049 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-16T09:33:58.049 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-16T09:33:58.298 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:33:58.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:58.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:58 smithi119 ceph-mon[35632]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:58.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:58.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:58.707 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:58.707 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:58 smithi077 ceph-mon[32171]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:58.707 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:58.707 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:58.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:33:58.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:58 smithi129 ceph-mon[35841]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:33:58.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:58.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.941+0000 7f8df1390700 1 -- 172.21.15.77:0/509744206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 msgr2=0x7f8dec1019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.941+0000 7f8df1390700 1 --2- 172.21.15.77:0/509744206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec1019e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009910 tx=0x7f8ddc009c20 comp rx=0 tx=0).stop 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.941+0000 7f8df1390700 1 -- 172.21.15.77:0/509744206 shutdown_connections 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.941+0000 7f8df1390700 1 --2- 172.21.15.77:0/509744206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec1019e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.942+0000 7f8df1390700 1 -- 172.21.15.77:0/509744206 >> 172.21.15.77:0/509744206 conn(0x7f8dec077280 msgr2=0x7f8dec077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.942+0000 7f8df1390700 1 -- 172.21.15.77:0/509744206 shutdown_connections 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.942+0000 7f8df1390700 1 -- 172.21.15.77:0/509744206 wait complete. 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.942+0000 7f8df1390700 1 Processor -- start 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.942+0000 7f8df1390700 1 -- start start 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.942+0000 7f8df1390700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec0fd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8df1390700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dec0fd880 0x7f8dec0fdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:59.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8df1390700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8dec1a3b80 0x7f8dec1a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8df1390700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec10b0c0 con 0x7f8dec101600 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8df1390700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec10af40 con 0x7f8dec1a3b80 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8df1390700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec10b240 con 0x7f8dec0fd880 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec0fd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec0fd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49862/0 (socket says 172.21.15.77:49862) 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 -- 172.21.15.77:0/453620472 learned_addr learned my addr 172.21.15.77:0/453620472 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8dea7fc700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dec0fd880 0x7f8dec0fdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 -- 172.21.15.77:0/453620472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dec0fd880 msgr2=0x7f8dec0fdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deb7fe700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8dec1a3b80 0x7f8dec1a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dec0fd880 0x7f8dec0fdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 -- 172.21.15.77:0/453620472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8dec1a3b80 msgr2=0x7f8dec1a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8dec1a3b80 0x7f8dec1a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 -- 172.21.15.77:0/453620472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec1a66a0 con 0x7f8dec101600 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deb7fe700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8dec1a3b80 0x7f8dec1a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:33:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8deaffd700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec0fd340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc005b40 tx=0x7f8ddc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8dd3fff700 1 -- 172.21.15.77:0/453620472 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc026070 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8dd3fff700 1 -- 172.21.15.77:0/453620472 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ddc00b6f0 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dec1a6990 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8dd3fff700 1 -- 172.21.15.77:0/453620472 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc0038e0 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.943+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dec1a6ed0 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.944+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dec063900 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.944+0000 7f8dd3fff700 1 -- 172.21.15.77:0/453620472 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f8ddc029070 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.945+0000 7f8dd3fff700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8dd4039450 0x7f8dd403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.945+0000 7f8dd3fff700 1 -- 172.21.15.77:0/453620472 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8ddc04ccc0 con 0x7f8dec101600 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.945+0000 7f8dea7fc700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8dd4039450 0x7f8dd403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.945+0000 7f8dea7fc700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8dd4039450 0x7f8dd403b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009850 tx=0x7f8de0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:33:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:33:59.950+0000 7f8dd3fff700 1 -- 172.21.15.77:0/453620472 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ddc013070 con 0x7f8dec101600 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:33:59.954 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:33:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:00.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.156+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8dec1a71b0 con 0x7f8dec101600 2024-03-16T09:34:00.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.157+0000 7f8dd3fff700 1 -- 172.21.15.77:0/453620472 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f8ddc030070 con 0x7f8dec101600 2024-03-16T09:34:00.160 INFO:teuthology.orchestra.run.smithi077.stdout:# minimal ceph.conf for e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:34:00.160 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-16T09:34:00.160 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:34:00.160 INFO:teuthology.orchestra.run.smithi077.stdout: mon_host = [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8dd4039450 msgr2=0x7f8dd403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8dd4039450 0x7f8dd403b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009850 tx=0x7f8de0008040 comp rx=0 tx=0).stop 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 msgr2=0x7f8dec0fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec0fd340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc005b40 tx=0x7f8ddc005030 comp rx=0 tx=0).stop 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 shutdown_connections 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8dd4039450 0x7f8dd403b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8dec1a3b80 0x7f8dec1a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dec101600 0x7f8dec0fd340 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 --2- 172.21.15.77:0/453620472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dec0fd880 0x7f8dec0fdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 >> 172.21.15.77:0/453620472 conn(0x7f8dec077280 msgr2=0x7f8dec0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 shutdown_connections 2024-03-16T09:34:00.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:00.159+0000 7f8df1390700 1 -- 172.21.15.77:0/453620472 wait complete. 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:33:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:00.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:00.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:00.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:00.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:33:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Updating smithi077:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Updating smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Updating smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T09:34:01.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: Reconfiguring daemon mon.a on smithi077 2024-03-16T09:34:01.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:01.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:00 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/453620472' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:01.185 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-16T09:34:01.185 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:34:01.185 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Updating smithi077:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Updating smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Updating smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: Reconfiguring daemon mon.a on smithi077 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:00 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/453620472' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:01.214 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:34:01.214 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:34:01.281 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:34:01.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T09:34:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T09:34:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Updating smithi077:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Updating smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Updating smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: Reconfiguring daemon mon.a on smithi077 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:00 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/453620472' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:01.316 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:34:01.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:34:01.387 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:34:01.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T09:34:01.421 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:34:01.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T09:34:01.492 INFO:tasks.cephadm:Adding mgr.x on smithi077 2024-03-16T09:34:01.492 INFO:tasks.cephadm:Adding mgr.y on smithi119 2024-03-16T09:34:01.492 INFO:tasks.cephadm:Adding mgr.z on smithi129 2024-03-16T09:34:01.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi077=x;smithi119=y;smithi129=z' 2024-03-16T09:34:01.765 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: Reconfiguring mon.b (monmap changed)... 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:02.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:02 smithi119 ceph-mon[35632]: Reconfiguring daemon mon.b on smithi119 2024-03-16T09:34:02.957 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:02.957 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:02.957 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:02.957 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: Reconfiguring mon.b (monmap changed)... 2024-03-16T09:34:02.958 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:02.958 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:02.958 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:02.958 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:02 smithi129 ceph-mon[35841]: Reconfiguring daemon mon.b on smithi119 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: Reconfiguring mon.b (monmap changed)... 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:02 smithi077 ceph-mon[32171]: Reconfiguring daemon mon.b on smithi119 2024-03-16T09:34:03.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.267+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/1282082920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64100cf0 msgr2=0x7f6b64101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.267+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/1282082920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64100cf0 0x7f6b64101110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009910 tx=0x7f6b60009c20 comp rx=0 tx=0).stop 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.267+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/1282082920 shutdown_connections 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.267+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/1282082920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64100cf0 0x7f6b64101110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.267+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/1282082920 >> 172.21.15.129:0/1282082920 conn(0x7f6b640fc2b0 msgr2=0x7f6b640fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/1282082920 shutdown_connections 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/1282082920 wait complete. 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 Processor -- start 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 -- start start 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b64100cf0 0x7f6b64196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64196990 0x7f6b6419ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:03.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b64196e10 0x7f6b6419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64106c30 con 0x7f6b64196990 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64106ab0 con 0x7f6b64196e10 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.268+0000 7f6b6b6e0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64106db0 con 0x7f6b64100cf0 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b64100cf0 0x7f6b64196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b64100cf0 0x7f6b64196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39388/0 (socket says 172.21.15.129:39388) 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 -- 172.21.15.129:0/197018077 learned_addr learned my addr 172.21.15.129:0/197018077 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b68c7b700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64196990 0x7f6b6419ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b69c7d700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b64196e10 0x7f6b6419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 -- 172.21.15.129:0/197018077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b64196e10 msgr2=0x7f6b6419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b64196e10 0x7f6b6419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 -- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64196990 msgr2=0x7f6b6419ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64196990 0x7f6b6419ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 -- 172.21.15.129:0/197018077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6419e860 con 0x7f6b64100cf0 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6947c700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b64100cf0 0x7f6b64196450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005e00 tx=0x7f6b60005020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:34:03.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b5a7fc700 1 -- 172.21.15.129:0/197018077 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b60026070 con 0x7f6b64100cf0 2024-03-16T09:34:03.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b5a7fc700 1 -- 172.21.15.129:0/197018077 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b6000b750 con 0x7f6b64100cf0 2024-03-16T09:34:03.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6419eaf0 con 0x7f6b64100cf0 2024-03-16T09:34:03.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b5a7fc700 1 -- 172.21.15.129:0/197018077 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b600038a0 con 0x7f6b64100cf0 2024-03-16T09:34:03.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.269+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6419f050 con 0x7f6b64100cf0 2024-03-16T09:34:03.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.270+0000 7f6b5a7fc700 1 -- 172.21.15.129:0/197018077 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f6b60029070 con 0x7f6b64100cf0 2024-03-16T09:34:03.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.270+0000 7f6b5a7fc700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b50039450 0x7f6b5003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:03.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.270+0000 7f6b5a7fc700 1 -- 172.21.15.129:0/197018077 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6b6004ccf0 con 0x7f6b64100cf0 2024-03-16T09:34:03.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.271+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b64063900 con 0x7f6b64100cf0 2024-03-16T09:34:03.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.271+0000 7f6b68c7b700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b50039450 0x7f6b5003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:03.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.271+0000 7f6b68c7b700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b50039450 0x7f6b5003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009ad0 tx=0x7f6b54009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:34:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.276+0000 7f6b5a7fc700 1 -- 172.21.15.129:0/197018077 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b60030070 con 0x7f6b64100cf0 2024-03-16T09:34:03.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.490+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi119=y;smithi129=z", "target": ["mon-mgr", ""]}) -- 0x7f6b64062710 con 0x7f6b50039450 2024-03-16T09:34:03.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.497+0000 7f6b5a7fc700 1 -- 172.21.15.129:0/197018077 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6b64062710 con 0x7f6b50039450 2024-03-16T09:34:03.501 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mgr update... 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b50039450 msgr2=0x7f6b5003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b50039450 0x7f6b5003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009ad0 tx=0x7f6b54009210 comp rx=0 tx=0).stop 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b64100cf0 msgr2=0x7f6b64196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b64100cf0 0x7f6b64196450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005e00 tx=0x7f6b60005020 comp rx=0 tx=0).stop 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 shutdown_connections 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b50039450 0x7f6b5003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b64196e10 0x7f6b6419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b64100cf0 0x7f6b64196450 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:03.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 --2- 172.21.15.129:0/197018077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b64196990 0x7f6b6419ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:03.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.499+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 >> 172.21.15.129:0/197018077 conn(0x7f6b640fc2b0 msgr2=0x7f6b640fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:34:03.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.500+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 shutdown_connections 2024-03-16T09:34:03.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:34:03.500+0000 7f6b6b6e0700 1 -- 172.21.15.129:0/197018077 wait complete. 2024-03-16T09:34:04.098 DEBUG:teuthology.orchestra.run.smithi119:mgr.y> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.y.service 2024-03-16T09:34:04.101 DEBUG:teuthology.orchestra.run.smithi129:mgr.z> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.z.service 2024-03-16T09:34:04.103 INFO:tasks.cephadm:Deploying OSDs... 2024-03-16T09:34:04.103 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:34:04.103 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T09:34:04.122 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T09:34:04.122 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 09:33:10.983328117 +0000 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 09:28:43.141355706 +0000 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 09:28:43.141355706 +0000 2024-03-16T09:34:04.181 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 09:28:43.141355706 +0000 2024-03-16T09:34:04.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T09:34:04.252 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T09:34:04.252 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T09:34:04.253 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000232648 s, 2.2 MB/s 2024-03-16T09:34:04.254 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T09:34:04.312 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 09:33:11.016327439 +0000 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 09:28:43.487348657 +0000 2024-03-16T09:34:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 09:28:43.487348657 +0000 2024-03-16T09:34:04.370 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 09:28:43.487348657 +0000 2024-03-16T09:34:04.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T09:34:04.438 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T09:34:04.438 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T09:34:04.438 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000216532 s, 2.4 MB/s 2024-03-16T09:34:04.439 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: Reconfiguring mon.c (monmap changed)... 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: Reconfiguring daemon mon.c on smithi129 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi119=y;smithi129=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: Saving service mgr spec with placement smithi077=x;smithi119=y;smithi129=z;count:3 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:04 smithi119 ceph-mon[35632]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:04.496 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: Reconfiguring mon.c (monmap changed)... 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: Reconfiguring daemon mon.c on smithi129 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi119=y;smithi129=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:04.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: Saving service mgr spec with placement smithi077=x;smithi119=y;smithi129=z;count:3 2024-03-16T09:34:04.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:04 smithi077 ceph-mon[32171]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:04.516 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.516 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.516 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: Reconfiguring mon.c (monmap changed)... 2024-03-16T09:34:04.516 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T09:34:04.516 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T09:34:04.516 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:04.517 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: Reconfiguring daemon mon.c on smithi129 2024-03-16T09:34:04.517 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi119=y;smithi129=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:04.517 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: Saving service mgr spec with placement smithi077=x;smithi119=y;smithi129=z;count:3 2024-03-16T09:34:04.517 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:04.517 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:04 smithi129 ceph-mon[35841]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:04.524 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T09:34:04.524 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:04.524 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-16T09:34:04.524 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:04.525 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:04.525 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 09:33:11.048326781 +0000 2024-03-16T09:34:04.525 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 09:28:43.834341588 +0000 2024-03-16T09:34:04.525 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 09:28:43.834341588 +0000 2024-03-16T09:34:04.525 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 09:28:43.834341588 +0000 2024-03-16T09:34:04.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T09:34:04.590 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T09:34:04.590 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T09:34:04.591 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000200938 s, 2.5 MB/s 2024-03-16T09:34:04.592 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T09:34:04.647 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 09:33:11.079326144 +0000 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 09:28:44.182334498 +0000 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 09:28:44.182334498 +0000 2024-03-16T09:34:04.703 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 09:28:44.182334498 +0000 2024-03-16T09:34:04.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T09:34:04.768 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T09:34:04.768 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T09:34:04.768 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000283363 s, 1.8 MB/s 2024-03-16T09:34:04.769 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T09:34:04.825 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-16T09:34:04.825 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T09:34:04.844 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T09:34:04.844 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-03-16T09:34:04.902 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T09:34:04.902 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:04.902 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-16T09:34:04.902 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:04.902 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:04.903 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-16 09:33:51.835440243 +0000 2024-03-16T09:34:04.903 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-16 09:28:43.832877663 +0000 2024-03-16T09:34:04.903 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-16 09:28:43.832877663 +0000 2024-03-16T09:34:04.903 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-16 09:28:43.832877663 +0000 2024-03-16T09:34:04.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T09:34:04.972 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-16T09:34:04.972 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-16T09:34:04.972 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000261534 s, 2.0 MB/s 2024-03-16T09:34:04.973 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T09:34:05.032 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-16 09:33:51.835440243 +0000 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-16 09:28:44.176869074 +0000 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-16 09:28:44.176869074 +0000 2024-03-16T09:34:05.088 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-16 09:28:44.176869074 +0000 2024-03-16T09:34:05.088 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T09:34:05.156 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-16T09:34:05.156 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-16T09:34:05.156 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000210622 s, 2.4 MB/s 2024-03-16T09:34:05.157 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T09:34:05.214 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-03-16T09:34:05.269 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T09:34:05.269 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:05.269 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-16T09:34:05.269 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:05.269 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:05.270 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-16 09:33:51.835440243 +0000 2024-03-16T09:34:05.270 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-16 09:28:44.524860385 +0000 2024-03-16T09:34:05.270 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-16 09:28:44.524860385 +0000 2024-03-16T09:34:05.270 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-16 09:28:44.524860385 +0000 2024-03-16T09:34:05.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T09:34:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-16T09:34:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-16T09:34:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000280177 s, 1.8 MB/s 2024-03-16T09:34:05.337 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T09:34:05.393 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-03-16T09:34:05.448 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T09:34:05.448 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:05.448 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-16T09:34:05.449 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:05.449 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:05.449 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-16 09:33:51.836440218 +0000 2024-03-16T09:34:05.449 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-16 09:28:44.874851646 +0000 2024-03-16T09:34:05.449 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-16 09:28:44.874851646 +0000 2024-03-16T09:34:05.449 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-16 09:28:44.874851646 +0000 2024-03-16T09:34:05.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T09:34:05.514 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-16T09:34:05.514 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-16T09:34:05.514 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000211424 s, 2.4 MB/s 2024-03-16T09:34:05.515 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T09:34:05.571 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-16T09:34:05.571 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T09:34:05.590 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T09:34:05.590 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-16 09:33:42.945396039 +0000 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-16 09:28:44.077275616 +0000 2024-03-16T09:34:05.647 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-16 09:28:44.077275616 +0000 2024-03-16T09:34:05.648 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-16 09:28:44.077275616 +0000 2024-03-16T09:34:05.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T09:34:05.717 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-16T09:34:05.717 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-16T09:34:05.717 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000220054 s, 2.3 MB/s 2024-03-16T09:34:05.718 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T09:34:05.774 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-16 09:33:42.945396039 +0000 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-16 09:28:44.411268262 +0000 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-16 09:28:44.411268262 +0000 2024-03-16T09:34:05.829 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-16 09:28:44.411268262 +0000 2024-03-16T09:34:05.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T09:34:05.895 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-16T09:34:05.895 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-16T09:34:05.895 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000232343 s, 2.2 MB/s 2024-03-16T09:34:05.897 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T09:34:05.952 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-03-16T09:34:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T09:34:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:06.008 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-16T09:34:06.008 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:06.008 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:06.008 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-16 09:33:42.945396039 +0000 2024-03-16T09:34:06.008 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-16 09:28:44.742260974 +0000 2024-03-16T09:34:06.009 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-16 09:28:44.742260974 +0000 2024-03-16T09:34:06.009 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-16 09:28:44.742260974 +0000 2024-03-16T09:34:06.009 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T09:34:06.074 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-16T09:34:06.074 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-16T09:34:06.074 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000197735 s, 2.6 MB/s 2024-03-16T09:34:06.075 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T09:34:06.130 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-03-16T09:34:06.185 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T09:34:06.185 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T09:34:06.185 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-16T09:34:06.185 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:34:06.186 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T09:34:06.186 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-16 09:33:42.946396017 +0000 2024-03-16T09:34:06.186 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-16 09:28:45.087253377 +0000 2024-03-16T09:34:06.186 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-16 09:28:45.087253377 +0000 2024-03-16T09:34:06.186 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-16 09:28:45.087253377 +0000 2024-03-16T09:34:06.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T09:34:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-16T09:34:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-16T09:34:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000216591 s, 2.4 MB/s 2024-03-16T09:34:06.254 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T09:34:06.310 INFO:tasks.cephadm:Deploying osd.0 on smithi077 with /dev/vg_nvme/lv_4... 2024-03-16T09:34:06.310 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:06.350 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:06.351 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:06 smithi129 ceph-mon[35841]: Deploying daemon mgr.z on smithi129 2024-03-16T09:34:06.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:06.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:06.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:06.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:06.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:06.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T09:34:06.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:06.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:06.442 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:06 smithi119 ceph-mon[35632]: Deploying daemon mgr.z on smithi129 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:06.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T09:34:06.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:06.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:06.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:06 smithi077 ceph-mon[32171]: Deploying daemon mgr.z on smithi129 2024-03-16T09:34:06.543 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:34:07.801 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:07 smithi129 systemd[1]: Starting Ceph mgr.z for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:34:08.205 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:07 smithi129 podman[37678]: 2024-03-16 09:34:07.909563618 +0000 UTC m=+0.102408006 container create bc3aef4b02e461e840f80d4911b15b0560f759c08df6f8a722f663ffe9df0631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:34:08.206 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:07 smithi129 podman[37678]: 2024-03-16 09:34:07.825297749 +0000 UTC m=+0.018142094 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:34:08.206 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:08 smithi129 podman[37678]: 2024-03-16 09:34:08.128205171 +0000 UTC m=+0.321049515 container init bc3aef4b02e461e840f80d4911b15b0560f759c08df6f8a722f663ffe9df0631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-16T09:34:08.206 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:08 smithi129 podman[37678]: 2024-03-16 09:34:08.130738431 +0000 UTC m=+0.323582777 container start bc3aef4b02e461e840f80d4911b15b0560f759c08df6f8a722f663ffe9df0631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T09:34:08.206 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mgr[37697]: -- 172.21.15.129:0/4241069635 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ff5a965a0 con 0x555ff5a6b000 2024-03-16T09:34:08.206 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:08 smithi129 bash[37678]: bc3aef4b02e461e840f80d4911b15b0560f759c08df6f8a722f663ffe9df0631 2024-03-16T09:34:08.444 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T09:34:08.445 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T09:34:08.445 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T09:34:08.445 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T09:34:08.445 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139887 s, 750 MB/s 2024-03-16T09:34:08.445 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T09:34:08.550 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:08 smithi129 systemd[1]: Started Ceph mgr.z for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:09.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:09.050 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:08.672+0000 7f96a1f11200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T09:34:09.050 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:08.766+0000 7f96a1f11200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:09.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-03-16T09:34:09.312 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:34:09.541 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:09.412+0000 7f96a1f11200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T09:34:09.541 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:09.507+0000 7f96a1f11200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T09:34:09.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:09 smithi129 ceph-mon[35841]: Deploying daemon mgr.y on smithi119 2024-03-16T09:34:09.801 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:09.656+0000 7f96a1f11200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T09:34:09.801 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:09.756+0000 7f96a1f11200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T09:34:10.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:09 smithi077 ceph-mon[32171]: Deploying daemon mgr.y on smithi119 2024-03-16T09:34:10.037 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:09 smithi119 ceph-mon[35632]: Deploying daemon mgr.y on smithi119 2024-03-16T09:34:10.117 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:09.842+0000 7f96a1f11200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T09:34:10.117 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:10.115+0000 7f96a1f11200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T09:34:10.412 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:10.226+0000 7f96a1f11200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T09:34:10.412 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:10.313+0000 7f96a1f11200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T09:34:10.412 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:10.409+0000 7f96a1f11200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T09:34:10.770 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:10.589+0000 7f96a1f11200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 -- 172.21.15.77:0/2750170190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103ea0 msgr2=0x7f6d5c10a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/2750170190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103ea0 0x7f6d5c10a750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400b0d0 tx=0x7f6d5400b3e0 comp rx=0 tx=0).stop 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 -- 172.21.15.77:0/2750170190 shutdown_connections 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/2750170190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d5c102300 0x7f6d5c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/2750170190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103ea0 0x7f6d5c10a750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/2750170190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d5c103500 0x7f6d5c103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 -- 172.21.15.77:0/2750170190 >> 172.21.15.77:0/2750170190 conn(0x7f6d5c0fd8e0 msgr2=0x7f6d5c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:34:10.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 -- 172.21.15.77:0/2750170190 shutdown_connections 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.799+0000 7f6d61bc8700 1 -- 172.21.15.77:0/2750170190 wait complete. 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 Processor -- start 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 -- start start 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d5c102300 0x7f6d5c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103500 0x7f6d5c198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d5c103ea0 0x7f6d5c19f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c10ddf0 con 0x7f6d5c103500 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c10dc70 con 0x7f6d5c103ea0 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d61bc8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c10df70 con 0x7f6d5c102300 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103500 0x7f6d5c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103500 0x7f6d5c198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60620/0 (socket says 172.21.15.77:60620) 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 -- 172.21.15.77:0/3186231065 learned_addr learned my addr 172.21.15.77:0/3186231065 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d5b7fe700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d5c102300 0x7f6d5c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:10.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d5bfff700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d5c103ea0 0x7f6d5c19f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 -- 172.21.15.77:0/3186231065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d5c102300 msgr2=0x7f6d5c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d5c102300 0x7f6d5c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 -- 172.21.15.77:0/3186231065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d5c103ea0 msgr2=0x7f6d5c19f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d5c103ea0 0x7f6d5c19f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.800+0000 7f6d52ffd700 1 -- 172.21.15.77:0/3186231065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5c1a0060 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.801+0000 7f6d52ffd700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103500 0x7f6d5c198940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400e9f0 tx=0x7f6d4400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.801+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400cbd0 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.801+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d4400ee90 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.801+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44005590 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.801+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5c1a0350 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.801+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5c1a0840 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.801+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f6d44021440 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.802+0000 7f6d597fa700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6d48039200 0x7f6d4803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.802+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6d4404c340 con 0x7f6d5c103500 2024-03-16T09:34:10.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.802+0000 7f6d5b7fe700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6d48039200 0x7f6d4803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.802+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5c02db70 con 0x7f6d5c103500 2024-03-16T09:34:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.802+0000 7f6d5b7fe700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6d48039200 0x7f6d4803b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c103360 tx=0x7f6d4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:34:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:10.807+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d44016bd0 con 0x7f6d5c103500 2024-03-16T09:34:10.896 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:10 smithi077 ceph-mon[32171]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:11.040+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6d5c062710 con 0x7f6d48039200 2024-03-16T09:34:11.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:10 smithi129 ceph-mon[35841]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:11.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:10 smithi119 ceph-mon[35632]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:11.190 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:10 smithi119 systemd[1]: Started Ceph mgr.y for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:34:11.690 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:11.252+0000 7f8812c16200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T09:34:11.690 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:11.349+0000 7f8812c16200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T09:34:12.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:12.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:34:12.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:34:12.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:11.625+0000 7f96a1f11200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:11.760+0000 7f96a1f11200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:34:12.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:12.121 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.024+0000 7f8812c16200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T09:34:12.121 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.118+0000 7f8812c16200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T09:34:12.121 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.121 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.121 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.121 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:12.121 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:12.122 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:34:12.122 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:34:12.122 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:12.300 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:12.107+0000 7f96a1f11200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T09:34:12.301 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:12.196+0000 7f96a1f11200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T09:34:12.439 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.263+0000 7f8812c16200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T09:34:12.440 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.362+0000 7f8812c16200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T09:34:12.713 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.446+0000 7f8812c16200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T09:34:12.713 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.710+0000 7f8812c16200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T09:34:12.800 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:12.415+0000 7f96a1f11200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:34:12.801 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:12.601+0000 7f96a1f11200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T09:34:13.001 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.819+0000 7f8812c16200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T09:34:13.001 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.905+0000 7f8812c16200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T09:34:13.001 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:12.998+0000 7f8812c16200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T09:34:13.001 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:13.002 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.144 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:12.881+0000 7f96a1f11200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:34:13.144 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:13.141+0000 7f96a1f11200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T09:34:13.144 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:13.145 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:13.439 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:13 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:13.172+0000 7f8812c16200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T09:34:13.550 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:13.376+0000 7f96a1f11200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T09:34:14.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:13 smithi119 ceph-mon[35632]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T09:34:14.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:13 smithi119 ceph-mon[35632]: Reconfiguring daemon mgr.x on smithi077 2024-03-16T09:34:14.132 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:13 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3915369300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6e86c30-0964-4336-a711-17623a2896cd"}]: dispatch 2024-03-16T09:34:14.133 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:13 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3915369300' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6e86c30-0964-4336-a711-17623a2896cd"}]': finished 2024-03-16T09:34:14.133 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:13 smithi119 ceph-mon[35632]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T09:34:14.133 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:14.174 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-mon[35841]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T09:34:14.174 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-mon[35841]: Reconfiguring daemon mgr.x on smithi077 2024-03-16T09:34:14.174 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3915369300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6e86c30-0964-4336-a711-17623a2896cd"}]: dispatch 2024-03-16T09:34:14.174 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3915369300' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6e86c30-0964-4336-a711-17623a2896cd"}]': finished 2024-03-16T09:34:14.174 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-mon[35841]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T09:34:14.177 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:13 smithi077 ceph-mon[32171]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T09:34:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:13 smithi077 ceph-mon[32171]: Reconfiguring daemon mgr.x on smithi077 2024-03-16T09:34:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:13 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3915369300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6e86c30-0964-4336-a711-17623a2896cd"}]: dispatch 2024-03-16T09:34:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:13 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3915369300' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6e86c30-0964-4336-a711-17623a2896cd"}]': finished 2024-03-16T09:34:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:13 smithi077 ceph-mon[32171]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T09:34:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:14.440 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:14.129+0000 7f8812c16200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T09:34:14.440 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:14.258+0000 7f8812c16200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T09:34:14.550 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:14.171+0000 7f96a1f11200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T09:34:14.844 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:14.588+0000 7f8812c16200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T09:34:14.845 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:14.675+0000 7f8812c16200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T09:34:15.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:14 smithi129 ceph-mon[35841]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:14 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3819610031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:34:15.051 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:14.738+0000 7f96a1f11200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T09:34:15.051 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:14.834+0000 7f96a1f11200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T09:34:15.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:14 smithi119 ceph-mon[35632]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:14 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3819610031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:34:15.190 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:14.868+0000 7f8812c16200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:34:15.190 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:15.050+0000 7f8812c16200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T09:34:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:14 smithi077 ceph-mon[32171]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:14 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3819610031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:34:15.550 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:34:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z[37693]: 2024-03-16T09:34:15.048+0000 7f96a1f11200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T09:34:15.587 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:15.330+0000 7f8812c16200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T09:34:15.588 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:15.584+0000 7f8812c16200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T09:34:15.850 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:15.818+0000 7f8812c16200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T09:34:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:15.852+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f6d440107b0 con 0x7f6d5c103500 2024-03-16T09:34:16.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-mon[35632]: Standby manager daemon z started 2024-03-16T09:34:16.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T09:34:16.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T09:34:16.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T09:34:16.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:15 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T09:34:16.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:15 smithi077 ceph-mon[32171]: Standby manager daemon z started 2024-03-16T09:34:16.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:15 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T09:34:16.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:15 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T09:34:16.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:15 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T09:34:16.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:15 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T09:34:16.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:15 smithi129 ceph-mon[35841]: Standby manager daemon z started 2024-03-16T09:34:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:15 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T09:34:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:15 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T09:34:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:15 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T09:34:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:15 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.129:0/3638027188' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T09:34:16.940 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:16 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:16.610+0000 7f8812c16200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T09:34:16.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:16 smithi119 ceph-mon[35632]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:16.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:16 smithi119 ceph-mon[35632]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-16T09:34:16.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T09:34:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:16 smithi077 ceph-mon[32171]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:16 smithi077 ceph-mon[32171]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-16T09:34:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T09:34:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:16 smithi129 ceph-mon[35841]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:16 smithi129 ceph-mon[35841]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-16T09:34:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T09:34:17.440 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:17.173+0000 7f8812c16200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T09:34:17.440 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:17.269+0000 7f8812c16200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T09:34:17.852 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y[37019]: 2024-03-16T09:34:17.480+0000 7f8812c16200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T09:34:17.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:17.860+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f6d4401f340 con 0x7f6d5c103500 2024-03-16T09:34:17.863 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:17 smithi077 ceph-mon[32171]: Standby manager daemon y started 2024-03-16T09:34:17.863 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:17 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T09:34:17.863 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:17 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T09:34:17.863 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:17 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T09:34:17.863 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:17 smithi077 ceph-mon[32171]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T09:34:18.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-mon[35632]: Standby manager daemon y started 2024-03-16T09:34:18.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T09:34:18.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T09:34:18.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T09:34:18.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:17 smithi119 ceph-mon[35632]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T09:34:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:17 smithi129 ceph-mon[35841]: Standby manager daemon y started 2024-03-16T09:34:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:17 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T09:34:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:17 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T09:34:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:17 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T09:34:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:17 smithi129 ceph-mon[35841]: from='mgr.? 172.21.15.119:0/2793283061' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T09:34:19.172 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:18 smithi077 ceph-mon[32171]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:19.172 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:18 smithi077 ceph-mon[32171]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-16T09:34:19.172 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T09:34:19.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:18 smithi119 ceph-mon[35632]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:19.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:18 smithi119 ceph-mon[35632]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-16T09:34:19.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T09:34:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:18 smithi129 ceph-mon[35841]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:18 smithi129 ceph-mon[35841]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-16T09:34:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T09:34:20.614 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:20.614 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:20.614 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:20.614 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:20.614 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:20.614 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:20.615 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:20 smithi077 ceph-mon[32171]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:20.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:20.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:20.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:20.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:20.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:20.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:20.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:20 smithi119 ceph-mon[35632]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:21.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:21.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:21.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:21.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:21.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:21.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:21.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:20 smithi129 ceph-mon[35841]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:22.796 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:22 smithi077 ceph-mon[32171]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:22.797 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T09:34:22.797 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:23.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:22 smithi129 ceph-mon[35841]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:23.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T09:34:23.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:23.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:22 smithi119 ceph-mon[35632]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:23.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T09:34:23.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:23.962 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:23 smithi077 ceph-mon[32171]: Deploying daemon osd.0 on smithi077 2024-03-16T09:34:24.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:23 smithi129 ceph-mon[35841]: Deploying daemon osd.0 on smithi077 2024-03-16T09:34:24.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:23 smithi119 ceph-mon[35632]: Deploying daemon osd.0 on smithi077 2024-03-16T09:34:24.847 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:24 smithi077 ceph-mon[32171]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:25.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:24 smithi129 ceph-mon[35841]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:25.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:24 smithi119 ceph-mon[35632]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:26 smithi077 ceph-mon[32171]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:27.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:26 smithi129 ceph-mon[35841]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:27.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:26 smithi119 ceph-mon[35632]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:28 smithi119 ceph-mon[35632]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:28 smithi077 ceph-mon[32171]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:28 smithi129 ceph-mon[35841]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:30.429 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.429 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.429 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:30.429 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:30.430 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.430 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:30 smithi077 ceph-mon[32171]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:30 smithi119 ceph-mon[35632]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:30.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:30.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:30.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:30.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:30 smithi129 ceph-mon[35841]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:31.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.113+0000 7f6d597fa700 1 -- 172.21.15.77:0/3186231065 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6d5c062710 con 0x7f6d48039200 2024-03-16T09:34:31.117 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 0 on host 'smithi077' 2024-03-16T09:34:31.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6d48039200 msgr2=0x7f6d4803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:31.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6d48039200 0x7f6d4803b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c103360 tx=0x7f6d4c009260 comp rx=0 tx=0).stop 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103500 msgr2=0x7f6d5c198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103500 0x7f6d5c198940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400e9f0 tx=0x7f6d4400ed00 comp rx=0 tx=0).stop 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 shutdown_connections 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6d48039200 0x7f6d4803b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d5c103ea0 0x7f6d5c19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d5c102300 0x7f6d5c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 --2- 172.21.15.77:0/3186231065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d5c103500 0x7f6d5c198940 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.115+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 >> 172.21.15.77:0/3186231065 conn(0x7f6d5c0fd8e0 msgr2=0x7f6d5c106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.116+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 shutdown_connections 2024-03-16T09:34:31.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:34:31.116+0000 7f6d61bc8700 1 -- 172.21.15.77:0/3186231065 wait complete. 2024-03-16T09:34:31.790 DEBUG:teuthology.orchestra.run.smithi077:osd.0> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.0.service 2024-03-16T09:34:31.792 INFO:tasks.cephadm:Deploying osd.1 on smithi119 with /dev/vg_nvme/lv_4... 2024-03-16T09:34:31.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T09:34:32.035 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:34:32.374 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:32.375 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:32.375 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:32.375 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:32 smithi119 ceph-mon[35632]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T09:34:32.375 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:32 smithi119 ceph-mon[35632]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:32.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:32.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:32.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:32.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:32 smithi077 ceph-mon[32171]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T09:34:32.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:32 smithi077 ceph-mon[32171]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:32.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:32.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:32.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:32.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:32 smithi129 ceph-mon[35841]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T09:34:32.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:32 smithi129 ceph-mon[35841]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:33.241 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:33 smithi077 ceph-mon[32171]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T09:34:33.241 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:33 smithi077 ceph-mon[32171]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T09:34:33.241 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:33 smithi077 ceph-mon[32171]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:34:33.241 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:33 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:33.242 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:34:33 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0[47393]: 2024-03-16T09:34:33.112+0000 7f4565f07700 -1 osd.0 0 waiting for initial osdmap 2024-03-16T09:34:33.242 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:34:33 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0[47393]: 2024-03-16T09:34:33.117+0000 7f4561cfa700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:34:33.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:33 smithi119 ceph-mon[35632]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T09:34:33.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:33 smithi119 ceph-mon[35632]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T09:34:33.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:33 smithi119 ceph-mon[35632]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:34:33.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:33 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:33.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:33 smithi129 ceph-mon[35841]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T09:34:33.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:33 smithi129 ceph-mon[35841]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T09:34:33.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:33 smithi129 ceph-mon[35841]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:34:33.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:33 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.023 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T09:34:34.023 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T09:34:34.023 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-16T09:34:34.023 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-16T09:34:34.023 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111921 s, 937 MB/s 2024-03-16T09:34:34.024 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 2024-03-16T09:34:34.024 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-16T09:34:34.327 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:34 smithi119 ceph-mon[35632]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:34:34.328 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:34 smithi119 ceph-mon[35632]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T09:34:34.328 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.328 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.328 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:34 smithi119 ceph-mon[35632]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:34.328 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.386 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:34 smithi077 ceph-mon[32171]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:34:34.386 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:34 smithi077 ceph-mon[32171]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T09:34:34.386 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.386 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.386 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:34 smithi077 ceph-mon[32171]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:34.386 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:34 smithi129 ceph-mon[35841]: from='osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:34:34.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:34 smithi129 ceph-mon[35841]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T09:34:34.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:34 smithi129 ceph-mon[35841]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T09:34:34.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:34.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-03-16T09:34:34.844 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:34:35.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:34:35.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:34:35.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] boot 2024-03-16T09:34:35.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T09:34:35.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi077 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi077 to 17088M 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:35.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:34:35.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:34:35.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] boot 2024-03-16T09:34:35.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi077 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi077 to 17088M 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: osd.0 [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] boot 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi077 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi077 to 17088M 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:35.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 -- 172.21.15.119:0/1097050522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103500 msgr2=0x7fabcc103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 --2- 172.21.15.119:0/1097050522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103500 0x7fabcc103960 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0099c0 tx=0x7fabbc009cd0 comp rx=0 tx=0).stop 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 -- 172.21.15.119:0/1097050522 shutdown_connections 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 --2- 172.21.15.119:0/1097050522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabcc103ea0 0x7fabcc10a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 --2- 172.21.15.119:0/1097050522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabcc102300 0x7fabcc102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 --2- 172.21.15.119:0/1097050522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103500 0x7fabcc103960 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 -- 172.21.15.119:0/1097050522 >> 172.21.15.119:0/1097050522 conn(0x7fabcc0fd8e0 msgr2=0x7fabcc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 -- 172.21.15.119:0/1097050522 shutdown_connections 2024-03-16T09:34:36.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 -- 172.21.15.119:0/1097050522 wait complete. 2024-03-16T09:34:36.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 Processor -- start 2024-03-16T09:34:36.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.361+0000 7fabd347f700 1 -- start start 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd347f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabcc102300 0x7fabcc1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd347f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabcc103500 0x7fabcc198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd347f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103ea0 0x7fabcc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd347f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc10ddf0 con 0x7fabcc102300 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd347f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc10dc70 con 0x7fabcc103ea0 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd347f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fabcc10df70 con 0x7fabcc103500 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103ea0 0x7fabcc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103ea0 0x7fabcc19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57060/0 (socket says 172.21.15.119:57060) 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 -- 172.21.15.119:0/1302250097 learned_addr learned my addr 172.21.15.119:0/1302250097 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd121b700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabcc102300 0x7fabcc1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd0a1a700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabcc103500 0x7fabcc198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 -- 172.21.15.119:0/1302250097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabcc103500 msgr2=0x7fabcc198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabcc103500 0x7fabcc198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:36.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 -- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabcc102300 msgr2=0x7fabcc1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabcc102300 0x7fabcc1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 -- 172.21.15.119:0/1302250097 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabcc1a0100 con 0x7fabcc103ea0 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd1a1c700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103ea0 0x7fabcc19fa60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fabc400ada0 tx=0x7fabc400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabc27fc700 1 -- 172.21.15.119:0/1302250097 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4017020 con 0x7fabcc103ea0 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabc27fc700 1 -- 172.21.15.119:0/1302250097 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabc400ce30 con 0x7fabcc103ea0 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.362+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabcc1a03f0 con 0x7fabcc103ea0 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.363+0000 7fabc27fc700 1 -- 172.21.15.119:0/1302250097 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4005460 con 0x7fabcc103ea0 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.363+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabcc1a0930 con 0x7fabcc103ea0 2024-03-16T09:34:36.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.363+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabcc068f90 con 0x7fabcc103ea0 2024-03-16T09:34:36.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.365+0000 7fabc27fc700 1 -- 172.21.15.119:0/1302250097 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fabc400c980 con 0x7fabcc103ea0 2024-03-16T09:34:36.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.365+0000 7fabc27fc700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabb80a2880 0x7fabb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:34:36.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.365+0000 7fabc27fc700 1 -- 172.21.15.119:0/1302250097 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fabc40ccd60 con 0x7fabcc103ea0 2024-03-16T09:34:36.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.365+0000 7fabd121b700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabb80a2880 0x7fabb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:34:36.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.366+0000 7fabd121b700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabb80a2880 0x7fabb80a4d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fabc8005fd0 tx=0x7fabc800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:34:36.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.369+0000 7fabc27fc700 1 -- 172.21.15.119:0/1302250097 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fabc4021020 con 0x7fabcc103ea0 2024-03-16T09:34:36.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:36 smithi119 ceph-mon[35632]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T09:34:36.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:36 smithi119 ceph-mon[35632]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:36.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:36 smithi077 ceph-mon[32171]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T09:34:36.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:36 smithi077 ceph-mon[32171]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:36.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:36 smithi129 ceph-mon[35841]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T09:34:36.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:36 smithi129 ceph-mon[35841]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:36.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:36.575+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fabcc062710 con 0x7fabb80a2880 2024-03-16T09:34:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:37 smithi119 ceph-mon[35632]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:34:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:34:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:37.452 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:37 smithi129 ceph-mon[35841]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:37.452 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:34:37.452 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:34:37.452 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:37.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:37 smithi077 ceph-mon[32171]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:34:37.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:34:37.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:34:37.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:38.418 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:38 smithi119 ceph-mon[35632]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:38.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:38 smithi077 ceph-mon[32171]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:38.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:38 smithi129 ceph-mon[35841]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:39 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/2750787347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]: dispatch 2024-03-16T09:34:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:39 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]: dispatch 2024-03-16T09:34:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:39 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]': finished 2024-03-16T09:34:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:39 smithi129 ceph-mon[35841]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T09:34:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:39 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:39.575 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:39 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/2750787347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]: dispatch 2024-03-16T09:34:39.575 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:39 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]: dispatch 2024-03-16T09:34:39.576 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:39 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]': finished 2024-03-16T09:34:39.576 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:39 smithi119 ceph-mon[35632]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T09:34:39.576 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:39 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:39.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:39 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/2750787347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]: dispatch 2024-03-16T09:34:39.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:39 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]: dispatch 2024-03-16T09:34:39.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:39 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03eeab1c-5a6a-4a70-a48d-df6a2d9183cb"}]': finished 2024-03-16T09:34:39.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:39 smithi077 ceph-mon[32171]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T09:34:39.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:39 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:40.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:40 smithi129 ceph-mon[35841]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:40.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:40 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/3530860879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:34:40.585 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:40 smithi119 ceph-mon[35632]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:40.585 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:40 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/3530860879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:34:40.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:40 smithi077 ceph-mon[32171]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:40.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:40 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/3530860879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:34:43.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:42 smithi129 ceph-mon[35841]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:43.115 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:42 smithi119 ceph-mon[35632]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:43.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:42 smithi077 ceph-mon[32171]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:45.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:44 smithi129 ceph-mon[35841]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:44 smithi119 ceph-mon[35632]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:45.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:44 smithi077 ceph-mon[32171]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:47.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:46 smithi129 ceph-mon[35841]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T09:34:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:47.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:46 smithi119 ceph-mon[35632]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T09:34:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:46 smithi077 ceph-mon[32171]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T09:34:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:48.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:47 smithi129 ceph-mon[35841]: Deploying daemon osd.1 on smithi119 2024-03-16T09:34:48.062 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:47 smithi119 ceph-mon[35632]: Deploying daemon osd.1 on smithi119 2024-03-16T09:34:48.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:47 smithi077 ceph-mon[32171]: Deploying daemon osd.1 on smithi119 2024-03-16T09:34:49.043 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:48 smithi119 ceph-mon[35632]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:49.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:48 smithi129 ceph-mon[35841]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:48 smithi077 ceph-mon[32171]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:50 smithi119 ceph-mon[35632]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:51.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:50 smithi129 ceph-mon[35841]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:50 smithi077 ceph-mon[32171]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:52.867 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:52 smithi119 ceph-mon[35632]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:52.867 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:52.867 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:52.867 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:53.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:52 smithi129 ceph-mon[35841]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:52 smithi077 ceph-mon[32171]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:55.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:54 smithi129 ceph-mon[35841]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:55.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:55.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:55.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:54 smithi119 ceph-mon[35632]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:54 smithi077 ceph-mon[32171]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:55.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.690+0000 7fabc27fc700 1 -- 172.21.15.119:0/1302250097 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fabcc062710 con 0x7fabb80a2880 2024-03-16T09:34:55.693 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 1 on host 'smithi119' 2024-03-16T09:34:55.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.692+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabb80a2880 msgr2=0x7fabb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.692+0000 7fabd347f700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabb80a2880 0x7fabb80a4d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fabc8005fd0 tx=0x7fabc800a560 comp rx=0 tx=0).stop 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.692+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103ea0 msgr2=0x7fabcc19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.692+0000 7fabd347f700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103ea0 0x7fabcc19fa60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fabc400ada0 tx=0x7fabc400c330 comp rx=0 tx=0).stop 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 shutdown_connections 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabb80a2880 0x7fabb80a4d40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabcc103ea0 0x7fabcc19fa60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabcc102300 0x7fabcc1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 --2- 172.21.15.119:0/1302250097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabcc103500 0x7fabcc198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 >> 172.21.15.119:0/1302250097 conn(0x7fabcc0fd8e0 msgr2=0x7fabcc106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 shutdown_connections 2024-03-16T09:34:55.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:34:55.693+0000 7fabd347f700 1 -- 172.21.15.119:0/1302250097 wait complete. 2024-03-16T09:34:56.322 DEBUG:teuthology.orchestra.run.smithi119:osd.1> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.1.service 2024-03-16T09:34:56.324 INFO:tasks.cephadm:Deploying osd.2 on smithi129 with /dev/vg_nvme/lv_4... 2024-03-16T09:34:56.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T09:34:56.568 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:34:56.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:56 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:56.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:56 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:56.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:56 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:56.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:56 smithi119 ceph-mon[35632]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:56.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:56 smithi119 ceph-mon[35632]: from='osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T09:34:56.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:56 smithi119 ceph-mon[35632]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T09:34:56.949 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:56 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:56.950 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:56 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:56.950 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:56 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:56.950 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:56 smithi129 ceph-mon[35841]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:56.950 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:56 smithi129 ceph-mon[35841]: from='osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T09:34:56.950 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:56 smithi129 ceph-mon[35841]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T09:34:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:56 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:56 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:34:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:56 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:56 smithi077 ceph-mon[32171]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:56 smithi077 ceph-mon[32171]: from='osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T09:34:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:56 smithi077 ceph-mon[32171]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T09:34:57.910 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:57 smithi119 ceph-mon[35632]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T09:34:57.910 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:57 smithi119 ceph-mon[35632]: from='osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:34:57.911 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:57 smithi119 ceph-mon[35632]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T09:34:57.911 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:57.911 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:57 smithi119 ceph-mon[35632]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:34:57.911 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:34:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1[40195]: 2024-03-16T09:34:57.707+0000 7f7378467700 -1 osd.1 0 waiting for initial osdmap 2024-03-16T09:34:57.911 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:34:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1[40195]: 2024-03-16T09:34:57.710+0000 7f737425a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:34:57.964 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:57 smithi129 ceph-mon[35841]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T09:34:57.964 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:57 smithi129 ceph-mon[35841]: from='osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:34:57.965 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:57 smithi129 ceph-mon[35841]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T09:34:57.965 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:57.965 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:57 smithi129 ceph-mon[35841]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:34:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:57 smithi077 ceph-mon[32171]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T09:34:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:57 smithi077 ceph-mon[32171]: from='osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:34:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:57 smithi077 ceph-mon[32171]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T09:34:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:58.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:57 smithi077 ceph-mon[32171]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:34:58.656 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T09:34:58.656 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T09:34:58.656 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-16T09:34:58.656 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-16T09:34:58.656 INFO:teuthology.orchestra.run.smithi129.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114516 s, 916 MB/s 2024-03-16T09:34:58.656 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-16T09:34:58.847 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:58 smithi119 ceph-mon[35632]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:34:58.847 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:58 smithi119 ceph-mon[35632]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T09:34:58.847 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:58.848 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:58.848 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:58 smithi119 ceph-mon[35632]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:58.848 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:58.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:58 smithi129 ceph-mon[35841]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:34:58.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:58 smithi129 ceph-mon[35841]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T09:34:58.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:58.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:58.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:58 smithi129 ceph-mon[35841]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:58.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:58 smithi077 ceph-mon[32171]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:34:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:58 smithi077 ceph-mon[32171]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T09:34:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:58 smithi077 ceph-mon[32171]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T09:34:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:59.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2024-03-16T09:34:59.536 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] boot 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:34:59.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:34:59.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:34:59.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:34:59.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:34:59 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:35:00.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:35:00.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] boot 2024-03-16T09:35:00.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:34:59 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:35:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:35:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: osd.1 [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] boot 2024-03-16T09:35:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T09:35:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T09:35:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:00.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:34:59 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:00.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:00 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi119 2024-03-16T09:35:00.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:00 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi119 to 17088M 2024-03-16T09:35:00.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:00 smithi129 ceph-mon[35841]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:00.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:00 smithi129 ceph-mon[35841]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T09:35:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:00 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi119 2024-03-16T09:35:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:00 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi119 to 17088M 2024-03-16T09:35:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:00 smithi077 ceph-mon[32171]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:00 smithi077 ceph-mon[32171]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T09:35:01.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 -- 172.21.15.129:0/2148642141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103500 msgr2=0x7f5a38103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:01.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2148642141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103500 0x7f5a38103960 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009990 tx=0x7f5a28009ca0 comp rx=0 tx=0).stop 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 -- 172.21.15.129:0/2148642141 shutdown_connections 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2148642141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a38102300 0x7f5a38102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2148642141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a38103ea0 0x7f5a3810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2148642141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103500 0x7f5a38103960 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 -- 172.21.15.129:0/2148642141 >> 172.21.15.129:0/2148642141 conn(0x7f5a380fd8e0 msgr2=0x7f5a380ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 -- 172.21.15.129:0/2148642141 shutdown_connections 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 -- 172.21.15.129:0/2148642141 wait complete. 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 Processor -- start 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.045+0000 7f5a404c3700 1 -- start start 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a404c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a38102300 0x7f5a381983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a404c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a38103500 0x7f5a38198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:01.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a404c3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103ea0 0x7f5a3819f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a404c3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3810ddd0 con 0x7f5a38102300 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a404c3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3810dc50 con 0x7f5a38103500 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a404c3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3810df50 con 0x7f5a38103ea0 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103ea0 0x7f5a3819f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103ea0 0x7f5a3819f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46770/0 (socket says 172.21.15.129:46770) 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 -- 172.21.15.129:0/2024848928 learned_addr learned my addr 172.21.15.129:0/2024848928 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3da5e700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a38103500 0x7f5a38198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3e25f700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a38102300 0x7f5a381983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 -- 172.21.15.129:0/2024848928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a38103500 msgr2=0x7f5a38198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a38103500 0x7f5a38198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 -- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a38102300 msgr2=0x7f5a381983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a38102300 0x7f5a381983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:01.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 -- 172.21.15.129:0/2024848928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a381a0050 con 0x7f5a38103ea0 2024-03-16T09:35:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a3ea60700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103ea0 0x7f5a3819f9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5a30007d40 tx=0x7f5a3000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:35:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a2f7fe700 1 -- 172.21.15.129:0/2024848928 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a30016020 con 0x7f5a38103ea0 2024-03-16T09:35:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a2f7fe700 1 -- 172.21.15.129:0/2024848928 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a3000acb0 con 0x7f5a38103ea0 2024-03-16T09:35:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a381a0340 con 0x7f5a38103ea0 2024-03-16T09:35:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.046+0000 7f5a2f7fe700 1 -- 172.21.15.129:0/2024848928 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a300115e0 con 0x7f5a38103ea0 2024-03-16T09:35:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.047+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a381a0830 con 0x7f5a38103ea0 2024-03-16T09:35:01.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.047+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a38068f90 con 0x7f5a38103ea0 2024-03-16T09:35:01.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.049+0000 7f5a2f7fe700 1 -- 172.21.15.129:0/2024848928 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f5a30019020 con 0x7f5a38103ea0 2024-03-16T09:35:01.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.049+0000 7f5a2f7fe700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5a240a2800 0x7f5a240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:01.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.049+0000 7f5a2f7fe700 1 -- 172.21.15.129:0/2024848928 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f5a300cd5c0 con 0x7f5a38103ea0 2024-03-16T09:35:01.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.049+0000 7f5a3e25f700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5a240a2800 0x7f5a240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:01.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.050+0000 7f5a3e25f700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5a240a2800 0x7f5a240a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a38103360 tx=0x7f5a3400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:35:01.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.053+0000 7f5a2f7fe700 1 -- 172.21.15.129:0/2024848928 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a30020020 con 0x7f5a38103ea0 2024-03-16T09:35:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:00 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi119 2024-03-16T09:35:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:00 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi119 to 17088M 2024-03-16T09:35:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:00 smithi119 ceph-mon[35632]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:00 smithi119 ceph-mon[35632]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T09:35:01.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:01.253+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5a38062710 con 0x7f5a240a2800 2024-03-16T09:35:02.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:02.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:02.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:02.029 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:01 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:02.029 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:01 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:02.029 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:01 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:01 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:01 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:01 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:02 smithi077 ceph-mon[32171]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:02 smithi077 ceph-mon[32171]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:03.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:02 smithi129 ceph-mon[35841]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:03.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:02 smithi129 ceph-mon[35841]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:02 smithi119 ceph-mon[35632]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:02 smithi119 ceph-mon[35632]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:04 smithi077 ceph-mon[32171]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:04 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/3678333197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]: dispatch 2024-03-16T09:35:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:04 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]: dispatch 2024-03-16T09:35:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:04 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]': finished 2024-03-16T09:35:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:04 smithi077 ceph-mon[32171]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T09:35:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:05.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:04 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/2141795994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:05.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:04 smithi129 ceph-mon[35841]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:05.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:04 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/3678333197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]: dispatch 2024-03-16T09:35:05.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:04 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]: dispatch 2024-03-16T09:35:05.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:04 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]': finished 2024-03-16T09:35:05.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:04 smithi129 ceph-mon[35841]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T09:35:05.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:05.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:04 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/2141795994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:04 smithi119 ceph-mon[35632]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:04 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/3678333197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]: dispatch 2024-03-16T09:35:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:04 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]: dispatch 2024-03-16T09:35:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:04 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7e55cef-7750-4ba5-ab01-45d97769a4fb"}]': finished 2024-03-16T09:35:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:04 smithi119 ceph-mon[35632]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T09:35:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:04 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/2141795994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:07.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:06 smithi129 ceph-mon[35841]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:07.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:06 smithi119 ceph-mon[35632]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:07.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:06 smithi077 ceph-mon[32171]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:09.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:08 smithi129 ceph-mon[35841]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:09.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:08 smithi119 ceph-mon[35632]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:09.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:08 smithi077 ceph-mon[32171]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:11.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:10 smithi129 ceph-mon[35841]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:11.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:10 smithi119 ceph-mon[35632]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:10 smithi077 ceph-mon[32171]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:12.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T09:35:12.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:12.188 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T09:35:12.188 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:12.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T09:35:12.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:12.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:12 smithi129 ceph-mon[35841]: Deploying daemon osd.2 on smithi129 2024-03-16T09:35:12.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:12 smithi129 ceph-mon[35841]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:13.160 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:12 smithi077 ceph-mon[32171]: Deploying daemon osd.2 on smithi129 2024-03-16T09:35:13.160 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:12 smithi077 ceph-mon[32171]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:13.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:12 smithi119 ceph-mon[35632]: Deploying daemon osd.2 on smithi129 2024-03-16T09:35:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:12 smithi119 ceph-mon[35632]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:14.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:14 smithi129 ceph-mon[35841]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:15.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:14 smithi119 ceph-mon[35632]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:14 smithi077 ceph-mon[32171]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:17.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:16 smithi129 ceph-mon[35841]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:17.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:17.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:17.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:16 smithi119 ceph-mon[35632]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:16 smithi077 ceph-mon[32171]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:19.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:18 smithi129 ceph-mon[35841]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:19.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:18 smithi119 ceph-mon[35632]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:18 smithi077 ceph-mon[32171]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:19.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.967+0000 7f5a2f7fe700 1 -- 172.21.15.129:0/2024848928 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5a38062710 con 0x7f5a240a2800 2024-03-16T09:35:19.970 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 2 on host 'smithi129' 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5a240a2800 msgr2=0x7f5a240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5a240a2800 0x7f5a240a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a38103360 tx=0x7f5a3400a380 comp rx=0 tx=0).stop 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103ea0 msgr2=0x7f5a3819f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103ea0 0x7f5a3819f9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5a30007d40 tx=0x7f5a3000a340 comp rx=0 tx=0).stop 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 shutdown_connections 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5a240a2800 0x7f5a240a4cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a38103ea0 0x7f5a3819f9b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a38102300 0x7f5a381983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 --2- 172.21.15.129:0/2024848928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a38103500 0x7f5a38198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 >> 172.21.15.129:0/2024848928 conn(0x7f5a380fd8e0 msgr2=0x7f5a38106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 shutdown_connections 2024-03-16T09:35:19.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:35:19.969+0000 7f5a404c3700 1 -- 172.21.15.129:0/2024848928 wait complete. 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:20.279 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:20.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:20.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:20.614 DEBUG:teuthology.orchestra.run.smithi129:osd.2> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.2.service 2024-03-16T09:35:20.617 INFO:tasks.cephadm:Deploying osd.3 on smithi077 with /dev/vg_nvme/lv_3... 2024-03-16T09:35:20.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T09:35:20.865 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:35:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:21 smithi077 ceph-mon[32171]: from='osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T09:35:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:21 smithi077 ceph-mon[32171]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T09:35:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:21 smithi129 ceph-mon[35841]: from='osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T09:35:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:21 smithi129 ceph-mon[35841]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T09:35:21.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:21 smithi119 ceph-mon[35632]: from='osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T09:35:21.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:21 smithi119 ceph-mon[35632]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T09:35:22.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-mon[35841]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T09:35:22.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-mon[35841]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T09:35:22.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:22.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-mon[35841]: from='osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:35:22.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-mon[35841]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:35:22.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-mon[35841]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:22.301 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2[40999]: 2024-03-16T09:35:22.150+0000 7f45fdcaa700 -1 osd.2 0 waiting for initial osdmap 2024-03-16T09:35:22.301 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2[40999]: 2024-03-16T09:35:22.153+0000 7f45f9a9d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:35:22.337 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:22 smithi077 ceph-mon[32171]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T09:35:22.337 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:22 smithi077 ceph-mon[32171]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T09:35:22.337 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:22.337 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:22 smithi077 ceph-mon[32171]: from='osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:35:22.337 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:22 smithi077 ceph-mon[32171]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:35:22.337 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:22 smithi077 ceph-mon[32171]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:22.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:22 smithi119 ceph-mon[35632]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T09:35:22.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:22 smithi119 ceph-mon[35632]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T09:35:22.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:22.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:22 smithi119 ceph-mon[35632]: from='osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:35:22.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:22 smithi119 ceph-mon[35632]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:35:22.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:22 smithi119 ceph-mon[35632]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:22.670 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T09:35:22.670 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T09:35:22.670 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T09:35:22.670 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T09:35:22.670 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129615 s, 809 MB/s 2024-03-16T09:35:22.670 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T09:35:23.307 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:23 smithi077 ceph-mon[32171]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:35:23.307 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:23 smithi077 ceph-mon[32171]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T09:35:23.307 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:23 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:23.307 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:23 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:23.333 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-03-16T09:35:23.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:23 smithi119 ceph-mon[35632]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:35:23.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:23 smithi119 ceph-mon[35632]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T09:35:23.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:23 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:23.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:23 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:23.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:23 smithi129 ceph-mon[35841]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:35:23.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:23 smithi129 ceph-mon[35841]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T09:35:23.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:23 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:23.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:23 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:23.583 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:35:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:35:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:35:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] boot 2024-03-16T09:35:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi129 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] boot 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi129 2024-03-16T09:35:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:24.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: osd.2 [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] boot 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi129 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-16T09:35:24.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.552 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:24.552 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:24.552 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:24.552 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T09:35:24.552 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 -- 172.21.15.77:0/716763334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c103f40 msgr2=0x7f989c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 --2- 172.21.15.77:0/716763334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c103f40 0x7f989c10a7f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f989400b0a0 tx=0x7f989400b3b0 comp rx=0 tx=0).stop 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 -- 172.21.15.77:0/716763334 shutdown_connections 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 --2- 172.21.15.77:0/716763334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1005c0 0x7f989c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 --2- 172.21.15.77:0/716763334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c103f40 0x7f989c10a7f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 --2- 172.21.15.77:0/716763334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989c1035a0 0x7f989c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 -- 172.21.15.77:0/716763334 >> 172.21.15.77:0/716763334 conn(0x7f989c0fbf40 msgr2=0x7f989c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 -- 172.21.15.77:0/716763334 shutdown_connections 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.225+0000 7f98a2c77700 1 -- 172.21.15.77:0/716763334 wait complete. 2024-03-16T09:35:25.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 Processor -- start 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 -- start start 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c1005c0 0x7f989c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1035a0 0x7f989c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989c103f40 0x7f989c19f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f989c10dd60 con 0x7f989c1005c0 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f989c10dbe0 con 0x7f989c1035a0 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a2c77700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f989c10dee0 con 0x7f989c103f40 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f98a0a13700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c1005c0 0x7f989c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f989bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1035a0 0x7f989c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f989bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1035a0 0x7f989c1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:51226/0 (socket says 172.21.15.77:51226) 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.226+0000 7f989bfff700 1 -- 172.21.15.77:0/791568900 learned_addr learned my addr 172.21.15.77:0/791568900 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f98a1214700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989c103f40 0x7f989c19f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f989bfff700 1 -- 172.21.15.77:0/791568900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989c103f40 msgr2=0x7f989c19f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f989bfff700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989c103f40 0x7f989c19f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f989bfff700 1 -- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c1005c0 msgr2=0x7f989c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:25.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f989bfff700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c1005c0 0x7f989c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f989bfff700 1 -- 172.21.15.77:0/791568900 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989c1a0000 con 0x7f989c1035a0 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f98a0a13700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c1005c0 0x7f989c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f98a1214700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989c103f40 0x7f989c19f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f989bfff700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1035a0 0x7f989c1988e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f988800e9a0 tx=0x7f988800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988800ca10 con 0x7f989c1035a0 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f989c1a02f0 con 0x7f989c1035a0 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f988800cbb0 con 0x7f989c1035a0 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.227+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888018680 con 0x7f989c1035a0 2024-03-16T09:35:25.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.228+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f989c1a0830 con 0x7f989c1035a0 2024-03-16T09:35:25.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.229+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f989c068f90 con 0x7f989c1035a0 2024-03-16T09:35:25.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.230+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f9888047a30 con 0x7f989c1035a0 2024-03-16T09:35:25.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.230+0000 7f9899ffb700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f988c0a2800 0x7f988c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:25.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.230+0000 7f98a0a13700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f988c0a2800 0x7f988c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:25.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.231+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f98880cd700 con 0x7f989c1035a0 2024-03-16T09:35:25.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.231+0000 7f98a0a13700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f988c0a2800 0x7f988c0a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f989c069d80 tx=0x7f9890009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:35:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.237+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f988809bce0 con 0x7f989c1035a0 2024-03-16T09:35:25.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:25 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T09:35:25.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:25 smithi119 ceph-mon[35632]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T09:35:25.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:25 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:35:25.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:25.449+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f989c0627d0 con 0x7f988c0a2800 2024-03-16T09:35:25.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:25 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T09:35:25.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:25 smithi077 ceph-mon[32171]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T09:35:25.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:25 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:35:25.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:25 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T09:35:25.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:25 smithi129 ceph-mon[35841]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T09:35:25.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:25 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:35:25.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[51993]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T09:35:25.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[51993]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:25.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[51993]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:25.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[51993]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:26.066 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[52012]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T09:35:26.066 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[52012]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:26.066 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[52012]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:26.067 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:35:25 smithi077 sudo[52012]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:26.304 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44015]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T09:35:26.304 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44015]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:26.304 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44015]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:26.304 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44015]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: from='client.24185 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:26.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:26.317 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T09:35:26.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T09:35:26.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: from='client.24185 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:26.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:26.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:26.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:26.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:26.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:26.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T09:35:26.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T09:35:26.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: from='client.24185 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:26.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:26.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:26.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:26.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:26.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:26.689 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44018]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T09:35:26.690 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44018]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:26.690 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44018]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:26.690 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:35:26 smithi119 sudo[44018]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:26.826 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T09:35:26.826 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:26.826 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44775]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:26.826 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44775]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:27.300 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T09:35:27.301 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:27.301 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44778]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:27.301 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:35:26 smithi129 sudo[44778]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:27.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52091]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T09:35:27.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52091]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:27.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52091]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:27.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52091]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:27.375+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9888097c50 con 0x7f989c1035a0 2024-03-16T09:35:27.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52115]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T09:35:27.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52115]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:27.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52115]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:27.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 sudo[52115]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:27.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 ceph-mon[32171]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T09:35:27.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:27.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:27.621 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:27 smithi077 ceph-mon[32171]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:27.621 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 ceph-mon[35841]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T09:35:27.621 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:27.621 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:27.621 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 ceph-mon[35841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:27.621 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T09:35:27.621 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:27.622 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44781]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:27.622 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44781]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:27.689 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:27 smithi119 ceph-mon[35632]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T09:35:27.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:27 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:27.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:27 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:27.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:27 smithi119 ceph-mon[35632]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:28.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44784]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T09:35:28.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44784]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:28.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44784]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:28.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:27 smithi129 sudo[44784]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:28.352 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44021]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T09:35:28.352 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44021]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:28.352 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44021]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:28.352 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44021]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44024]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T09:35:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44024]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T09:35:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44024]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T09:35:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 sudo[44024]: pam_unix(sudo:session): session closed for user root 2024-03-16T09:35:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/413398540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3207723-0545-48aa-8c7d-468dbc1b316a"}]: dispatch 2024-03-16T09:35:28.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/413398540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3207723-0545-48aa-8c7d-468dbc1b316a"}]': finished 2024-03-16T09:35:28.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T09:35:28.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:28.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/413398540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3207723-0545-48aa-8c7d-468dbc1b316a"}]: dispatch 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/413398540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3207723-0545-48aa-8c7d-468dbc1b316a"}]': finished 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T09:35:28.717 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:28.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/413398540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3207723-0545-48aa-8c7d-468dbc1b316a"}]: dispatch 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/413398540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3207723-0545-48aa-8c7d-468dbc1b316a"}]': finished 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T09:35:28.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:29.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:29 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2517472377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:29.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:29 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2517472377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:29.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:29 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2517472377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:30 smithi119 ceph-mon[35632]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:30 smithi077 ceph-mon[32171]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:30.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:30 smithi129 ceph-mon[35841]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:32.875 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:32 smithi077 ceph-mon[32171]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:33.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:32 smithi129 ceph-mon[35841]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:32 smithi119 ceph-mon[35632]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:35.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:34 smithi077 ceph-mon[32171]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:35.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:34 smithi129 ceph-mon[35841]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:34 smithi119 ceph-mon[35632]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T09:35:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:36.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T09:35:36.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:36.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T09:35:36.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:36.922 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:36 smithi077 ceph-mon[32171]: Deploying daemon osd.3 on smithi077 2024-03-16T09:35:36.922 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:36 smithi077 ceph-mon[32171]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:37.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:36 smithi129 ceph-mon[35841]: Deploying daemon osd.3 on smithi077 2024-03-16T09:35:37.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:36 smithi129 ceph-mon[35841]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:37.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:36 smithi119 ceph-mon[35632]: Deploying daemon osd.3 on smithi077 2024-03-16T09:35:37.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:36 smithi119 ceph-mon[35632]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:38.972 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:38 smithi077 ceph-mon[32171]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:39.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:38 smithi129 ceph-mon[35841]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:39.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:38 smithi119 ceph-mon[35632]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:41.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:40 smithi077 ceph-mon[32171]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:41.047 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:40 smithi129 ceph-mon[35841]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:41.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:40 smithi119 ceph-mon[35632]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:42.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:42.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:42.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:42.164 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:41 smithi077 ceph-mon[32171]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:42.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:42.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:42.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:42.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:41 smithi119 ceph-mon[35632]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:42.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:42.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:42.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:42.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:41 smithi129 ceph-mon[35841]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:44.167 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 3 on host 'smithi077' 2024-03-16T09:35:44.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.164+0000 7f9899ffb700 1 -- 172.21.15.77:0/791568900 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f989c0627d0 con 0x7f988c0a2800 2024-03-16T09:35:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f988c0a2800 msgr2=0x7f988c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f988c0a2800 0x7f988c0a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f989c069d80 tx=0x7f9890009260 comp rx=0 tx=0).stop 2024-03-16T09:35:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1035a0 msgr2=0x7f989c1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1035a0 0x7f989c1988e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f988800e9a0 tx=0x7f988800ecb0 comp rx=0 tx=0).stop 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 shutdown_connections 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f988c0a2800 0x7f988c0a4cc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f989c103f40 0x7f989c19f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f989c1005c0 0x7f989c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 --2- 172.21.15.77:0/791568900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f989c1035a0 0x7f989c1988e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.166+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 >> 172.21.15.77:0/791568900 conn(0x7f989c0fbf40 msgr2=0x7f989c1068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.167+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 shutdown_connections 2024-03-16T09:35:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:35:44.167+0000 7f98a2c77700 1 -- 172.21.15.77:0/791568900 wait complete. 2024-03-16T09:35:44.835 DEBUG:teuthology.orchestra.run.smithi077:osd.3> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.3.service 2024-03-16T09:35:44.838 INFO:tasks.cephadm:Deploying osd.4 on smithi119 with /dev/vg_nvme/lv_3... 2024-03-16T09:35:44.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T09:35:45.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:44 smithi129 ceph-mon[35841]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:45.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:44 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:45.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:44 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:45.082 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:35:45.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:44 smithi119 ceph-mon[35632]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:44 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:44 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:45.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:44 smithi077 ceph-mon[32171]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:45.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:44 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:45.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:44 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:45.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:45 smithi119 ceph-mon[35632]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T09:35:45.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:45 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:45.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:45 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:45 smithi077 ceph-mon[32171]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T09:35:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:45 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:45 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:46.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:45 smithi129 ceph-mon[35841]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T09:35:46.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:45 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:46.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:45 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:46.988 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T09:35:46.988 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T09:35:46.988 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-16T09:35:46.988 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-16T09:35:46.988 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112377 s, 933 MB/s 2024-03-16T09:35:46.988 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-16T09:35:46.989 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:46 smithi119 ceph-mon[35632]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:46.989 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:46 smithi119 ceph-mon[35632]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T09:35:46.989 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:46 smithi119 ceph-mon[35632]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T09:35:46.989 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:46 smithi119 ceph-mon[35632]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:35:46.989 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:46 smithi077 ceph-mon[32171]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:46 smithi077 ceph-mon[32171]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T09:35:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:46 smithi077 ceph-mon[32171]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T09:35:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:46 smithi077 ceph-mon[32171]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:35:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:47.010 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:35:46 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:35:46.788+0000 7f31a8b18700 -1 osd.3 0 waiting for initial osdmap 2024-03-16T09:35:47.010 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:35:46 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:35:46.793+0000 7f31a490b700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:35:47.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:46 smithi129 ceph-mon[35841]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:46 smithi129 ceph-mon[35841]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T09:35:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:46 smithi129 ceph-mon[35841]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T09:35:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:46 smithi129 ceph-mon[35841]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:35:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:47.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-03-16T09:35:47.821 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:35:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:47 smithi119 ceph-mon[35632]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:35:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:47 smithi119 ceph-mon[35632]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T09:35:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:47 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:47 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:47 smithi119 ceph-mon[35632]: osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] boot 2024-03-16T09:35:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:47 smithi119 ceph-mon[35632]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T09:35:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:47 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:48.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:47 smithi077 ceph-mon[32171]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:35:48.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:47 smithi077 ceph-mon[32171]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T09:35:48.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:47 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:48.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:47 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:48.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:47 smithi077 ceph-mon[32171]: osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] boot 2024-03-16T09:35:48.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:47 smithi077 ceph-mon[32171]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T09:35:48.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:47 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:48.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:47 smithi129 ceph-mon[35841]: from='osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:35:48.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:47 smithi129 ceph-mon[35841]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T09:35:48.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:47 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:48.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:47 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:48.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:47 smithi129 ceph-mon[35841]: osd.3 [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] boot 2024-03-16T09:35:48.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:47 smithi129 ceph-mon[35841]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T09:35:48.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:47 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T09:35:48.864 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:35:48.864 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:35:48.864 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:48.865 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:48.866 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:48.866 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:48.866 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:48 smithi119 ceph-mon[35632]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:49.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:49.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:49.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:49.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:48 smithi129 ceph-mon[35841]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T09:35:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:35:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:35:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:35:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:48 smithi077 ceph-mon[32171]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T09:35:49.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 -- 172.21.15.119:0/2623514472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 msgr2=0x7f30b40751b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:49.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 --2- 172.21.15.119:0/2623514472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b40751b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f30a8009990 tx=0x7f30a8009ca0 comp rx=0 tx=0).stop 2024-03-16T09:35:49.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 -- 172.21.15.119:0/2623514472 shutdown_connections 2024-03-16T09:35:49.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 --2- 172.21.15.119:0/2623514472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30b410a210 0x7f30b410c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:49.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 --2- 172.21.15.119:0/2623514472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30b4076300 0x7f30b4074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:49.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 --2- 172.21.15.119:0/2623514472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b40751b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:49.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 -- 172.21.15.119:0/2623514472 >> 172.21.15.119:0/2623514472 conn(0x7f30b40fd920 msgr2=0x7f30b40ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:35:49.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 -- 172.21.15.119:0/2623514472 shutdown_connections 2024-03-16T09:35:49.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.375+0000 7f30ba85f700 1 -- 172.21.15.119:0/2623514472 wait complete. 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 Processor -- start 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 -- start start 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30b4076300 0x7f30b419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30b410a210 0x7f30b41a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30b410ffe0 con 0x7f30b4076300 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f30b410fe60 con 0x7f30b4074d30 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30ba85f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30b4110160 con 0x7f30b410a210 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b419a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37676/0 (socket says 172.21.15.119:37676) 2024-03-16T09:35:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 -- 172.21.15.119:0/1882207561 learned_addr learned my addr 172.21.15.119:0/1882207561 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b8dfc700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30b410a210 0x7f30b41a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b37fe700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30b4076300 0x7f30b419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 -- 172.21.15.119:0/1882207561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30b410a210 msgr2=0x7f30b41a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30b410a210 0x7f30b41a1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 -- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30b4076300 msgr2=0x7f30b419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30b4076300 0x7f30b419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b3fff700 1 -- 172.21.15.119:0/1882207561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b41a2270 con 0x7f30b4074d30 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.376+0000 7f30b37fe700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30b4076300 0x7f30b419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.377+0000 7f30b3fff700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b419a610 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f30a400c8f0 tx=0x7f30a400cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.377+0000 7f30b17fa700 1 -- 172.21.15.119:0/1882207561 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4007940 con 0x7f30b4074d30 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.377+0000 7f30b17fa700 1 -- 172.21.15.119:0/1882207561 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30a400f450 con 0x7f30b4074d30 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.377+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b41a2560 con 0x7f30b4074d30 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.377+0000 7f30b17fa700 1 -- 172.21.15.119:0/1882207561 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4018660 con 0x7f30b4074d30 2024-03-16T09:35:49.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.377+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b41a2aa0 con 0x7f30b4074d30 2024-03-16T09:35:49.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.378+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30b4068f90 con 0x7f30b4074d30 2024-03-16T09:35:49.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.380+0000 7f30b17fa700 1 -- 172.21.15.119:0/1882207561 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f30a4007ae0 con 0x7f30b4074d30 2024-03-16T09:35:49.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.380+0000 7f30b17fa700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f309c0a2890 0x7f309c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:35:49.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.380+0000 7f30b17fa700 1 -- 172.21.15.119:0/1882207561 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3698+0+0 (secure 0 0 0) 0x7f30a40cd770 con 0x7f30b4074d30 2024-03-16T09:35:49.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.380+0000 7f30b37fe700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f309c0a2890 0x7f309c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:35:49.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.381+0000 7f30b37fe700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f309c0a2890 0x7f309c0a4d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f30a80038b0 tx=0x7f30a8003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:35:49.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.384+0000 7f30b17fa700 1 -- 172.21.15.119:0/1882207561 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30a4014030 con 0x7f30b4074d30 2024-03-16T09:35:49.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:35:49.587+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f30b4062710 con 0x7f309c0a2890 2024-03-16T09:35:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:49 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi077 2024-03-16T09:35:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:49 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi077 to 8544M 2024-03-16T09:35:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:49 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:49 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:49 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:49 smithi119 ceph-mon[35632]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T09:35:50.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:49 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi077 2024-03-16T09:35:50.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:49 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi077 to 8544M 2024-03-16T09:35:50.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:49 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:50.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:49 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:50.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:49 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:50.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:49 smithi129 ceph-mon[35841]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T09:35:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:49 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi077 2024-03-16T09:35:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:49 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi077 to 8544M 2024-03-16T09:35:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:49 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:35:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:49 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:35:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:49 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:35:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:49 smithi077 ceph-mon[32171]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T09:35:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:50 smithi119 ceph-mon[35632]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:50 smithi119 ceph-mon[35632]: pgmap v103: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:50 smithi119 ceph-mon[35632]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-16T09:35:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:50 smithi119 ceph-mon[35632]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T09:35:51.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:50 smithi129 ceph-mon[35841]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:51.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:50 smithi129 ceph-mon[35841]: pgmap v103: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:51.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:50 smithi129 ceph-mon[35841]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-16T09:35:51.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:50 smithi129 ceph-mon[35841]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T09:35:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:50 smithi077 ceph-mon[32171]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:35:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:50 smithi077 ceph-mon[32171]: pgmap v103: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:50 smithi077 ceph-mon[32171]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-16T09:35:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:50 smithi077 ceph-mon[32171]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T09:35:52.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:52 smithi119 ceph-mon[35632]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T09:35:52.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:52 smithi119 ceph-mon[35632]: pgmap v106: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:52.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:52 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/2699851415' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]: dispatch 2024-03-16T09:35:52.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:52 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]: dispatch 2024-03-16T09:35:52.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:52 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]': finished 2024-03-16T09:35:52.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:52 smithi119 ceph-mon[35632]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T09:35:52.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:35:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:52 smithi077 ceph-mon[32171]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T09:35:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:52 smithi077 ceph-mon[32171]: pgmap v106: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:52 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/2699851415' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]: dispatch 2024-03-16T09:35:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:52 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]: dispatch 2024-03-16T09:35:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:52 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]': finished 2024-03-16T09:35:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:52 smithi077 ceph-mon[32171]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T09:35:53.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:35:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:52 smithi129 ceph-mon[35841]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T09:35:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:52 smithi129 ceph-mon[35841]: pgmap v106: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:52 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/2699851415' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]: dispatch 2024-03-16T09:35:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:52 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]: dispatch 2024-03-16T09:35:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:52 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb"}]': finished 2024-03-16T09:35:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:52 smithi129 ceph-mon[35841]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T09:35:53.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:35:53.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:53 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/1904947980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:53.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:53 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/1904947980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:54.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:53 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/1904947980' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:35:54.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:54 smithi119 ceph-mon[35632]: pgmap v108: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:55.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:54 smithi077 ceph-mon[32171]: pgmap v108: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:55.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:54 smithi129 ceph-mon[35841]: pgmap v108: 1 pgs: 1 active+recovering+undersized+degraded+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 2/6 objects misplaced (33.333%) 2024-03-16T09:35:56.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:55 smithi119 ceph-mon[35632]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-16T09:35:56.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:55 smithi119 ceph-mon[35632]: Cluster is now healthy 2024-03-16T09:35:56.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:55 smithi077 ceph-mon[32171]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-16T09:35:56.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:55 smithi077 ceph-mon[32171]: Cluster is now healthy 2024-03-16T09:35:56.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:55 smithi129 ceph-mon[35841]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-16T09:35:56.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:55 smithi129 ceph-mon[35841]: Cluster is now healthy 2024-03-16T09:35:56.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:56 smithi119 ceph-mon[35632]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T09:35:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:56 smithi077 ceph-mon[32171]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T09:35:57.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:56 smithi129 ceph-mon[35841]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T09:35:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:35:58 smithi119 ceph-mon[35632]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:35:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:35:58 smithi077 ceph-mon[32171]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:35:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:35:58 smithi129 ceph-mon[35841]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:36:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:00 smithi119 ceph-mon[35632]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T09:36:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:00 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T09:36:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:00 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:00 smithi077 ceph-mon[32171]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T09:36:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:00 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T09:36:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:00 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:00 smithi129 ceph-mon[35841]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T09:36:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:00 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T09:36:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:00 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:01.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:01 smithi119 ceph-mon[35632]: Deploying daemon osd.4 on smithi119 2024-03-16T09:36:02.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:01 smithi077 ceph-mon[32171]: Deploying daemon osd.4 on smithi119 2024-03-16T09:36:02.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:01 smithi129 ceph-mon[35841]: Deploying daemon osd.4 on smithi119 2024-03-16T09:36:02.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:02 smithi119 ceph-mon[35632]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:36:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:02 smithi077 ceph-mon[32171]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:36:03.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:02 smithi129 ceph-mon[35841]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:36:05.018 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:04 smithi119 ceph-mon[35632]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-16T09:36:05.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:04 smithi077 ceph-mon[32171]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-16T09:36:05.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:04 smithi129 ceph-mon[35841]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-16T09:36:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:06 smithi119 ceph-mon[35632]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:36:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:06 smithi077 ceph-mon[32171]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:36:07.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:07.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:07.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:07.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:06 smithi129 ceph-mon[35841]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:36:08.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.801+0000 7f30b17fa700 1 -- 172.21.15.119:0/1882207561 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f30b4062710 con 0x7f309c0a2890 2024-03-16T09:36:08.804 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 4 on host 'smithi119' 2024-03-16T09:36:08.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.804+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f309c0a2890 msgr2=0x7f309c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:08.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.804+0000 7f30ba85f700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f309c0a2890 0x7f309c0a4d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f30a80038b0 tx=0x7f30a8003680 comp rx=0 tx=0).stop 2024-03-16T09:36:08.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.804+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 msgr2=0x7f30b419a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.804+0000 7f30ba85f700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b419a610 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f30a400c8f0 tx=0x7f30a400cc00 comp rx=0 tx=0).stop 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 shutdown_connections 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f309c0a2890 0x7f309c0a4d50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b4074d30 0x7f30b419a610 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30b410a210 0x7f30b41a1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 --2- 172.21.15.119:0/1882207561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30b4076300 0x7f30b419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 >> 172.21.15.119:0/1882207561 conn(0x7f30b40fd920 msgr2=0x7f30b4106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 shutdown_connections 2024-03-16T09:36:08.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:36:08.805+0000 7f30ba85f700 1 -- 172.21.15.119:0/1882207561 wait complete. 2024-03-16T09:36:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:08 smithi129 ceph-mon[35841]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:09.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:08 smithi119 ceph-mon[35632]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:09.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:08 smithi077 ceph-mon[32171]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:09.518 DEBUG:teuthology.orchestra.run.smithi119:osd.4> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.4.service 2024-03-16T09:36:09.521 INFO:tasks.cephadm:Deploying osd.5 on smithi129 with /dev/vg_nvme/lv_3... 2024-03-16T09:36:09.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T09:36:09.763 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:36:09.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:09 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:09.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:09 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:09.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:09 smithi129 ceph-mon[35841]: from='osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T09:36:09.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:09 smithi129 ceph-mon[35841]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T09:36:09.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:09 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:09.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:09 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:09.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:09 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:09.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:09 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:09.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:09 smithi119 ceph-mon[35632]: from='osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T09:36:09.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:09 smithi119 ceph-mon[35632]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T09:36:09.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:09 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:09.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:09 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:10.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:09 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:10.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:09 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:10.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:09 smithi077 ceph-mon[32171]: from='osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T09:36:10.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:09 smithi077 ceph-mon[32171]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T09:36:10.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:09 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:10.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:09 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:11.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:10 smithi129 ceph-mon[35841]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:11.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:10 smithi129 ceph-mon[35841]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T09:36:11.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:10 smithi129 ceph-mon[35841]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T09:36:11.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:10 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:11.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:10 smithi129 ceph-mon[35841]: from='osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:36:11.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:10 smithi129 ceph-mon[35841]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:36:11.100 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:10 smithi119 ceph-mon[35632]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:11.100 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:10 smithi119 ceph-mon[35632]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T09:36:11.100 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:10 smithi119 ceph-mon[35632]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T09:36:11.100 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:10 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:11.100 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:10 smithi119 ceph-mon[35632]: from='osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:36:11.100 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:10 smithi119 ceph-mon[35632]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:36:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:10 smithi077 ceph-mon[32171]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:10 smithi077 ceph-mon[32171]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T09:36:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:10 smithi077 ceph-mon[32171]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T09:36:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:10 smithi077 ceph-mon[32171]: from='osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:36:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:10 smithi077 ceph-mon[32171]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:36:11.637 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T09:36:11.637 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T09:36:11.637 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-16T09:36:11.637 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-16T09:36:11.637 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114219 s, 918 MB/s 2024-03-16T09:36:11.637 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-16T09:36:11.810 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:36:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4[46852]: 2024-03-16T09:36:11.737+0000 7f08a8581700 -1 osd.4 0 waiting for initial osdmap 2024-03-16T09:36:12.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:11 smithi119 ceph-mon[35632]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:36:12.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:11 smithi119 ceph-mon[35632]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T09:36:12.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:11 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:11 smithi119 ceph-mon[35632]: from='osd.4 ' entity='osd.4' 2024-03-16T09:36:12.190 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:36:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4[46852]: 2024-03-16T09:36:11.935+0000 7f089fb6b700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:36:12.201 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:11 smithi129 ceph-mon[35841]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:36:12.201 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:11 smithi129 ceph-mon[35841]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T09:36:12.201 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.201 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:11 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.201 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:11 smithi129 ceph-mon[35841]: from='osd.4 ' entity='osd.4' 2024-03-16T09:36:12.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2024-03-16T09:36:12.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:11 smithi077 ceph-mon[32171]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:36:12.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:11 smithi077 ceph-mon[32171]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T09:36:12.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:11 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:11 smithi077 ceph-mon[32171]: from='osd.4 ' entity='osd.4' 2024-03-16T09:36:12.463 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:36:12.871 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:12 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:36:12.871 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:12 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:36:12.872 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:12 smithi129 ceph-mon[35841]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:12.872 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.872 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:12 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:36:12.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:12 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:36:12.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:12 smithi119 ceph-mon[35632]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:12.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:12.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:13.160 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:12 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:36:13.160 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:12 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:36:13.160 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:12 smithi077 ceph-mon[32171]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:13.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:13.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] boot 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:13.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:13.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:13.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 -- 172.21.15.129:0/553899090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 msgr2=0x7fb08c1051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 --2- 172.21.15.129:0/553899090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c1051e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb08800b0a0 tx=0x7fb08800b3b0 comp rx=0 tx=0).stop 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 -- 172.21.15.129:0/553899090 shutdown_connections 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 --2- 172.21.15.129:0/553899090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c108050 0x7fb08c10a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 --2- 172.21.15.129:0/553899090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c1051e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 --2- 172.21.15.129:0/553899090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08c105720 0x7fb08c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 -- 172.21.15.129:0/553899090 >> 172.21.15.129:0/553899090 conn(0x7fb08c0fc7a0 msgr2=0x7fb08c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 -- 172.21.15.129:0/553899090 shutdown_connections 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.985+0000 7fb092f08700 1 -- 172.21.15.129:0/553899090 wait complete. 2024-03-16T09:36:13.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 Processor -- start 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 -- start start 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c105720 0x7fb08c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08c108050 0x7fb08c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10ddc0 con 0x7fb08c105720 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10dc40 con 0x7fb08c108050 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb092f08700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10df40 con 0x7fb08c102df0 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52356/0 (socket says 172.21.15.129:52356) 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 -- 172.21.15.129:0/3557826601 learned_addr learned my addr 172.21.15.129:0/3557826601 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb083fff700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c105720 0x7fb08c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb0914a5700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08c108050 0x7fb08c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 -- 172.21.15.129:0/3557826601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08c108050 msgr2=0x7fb08c19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08c108050 0x7fb08c19fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:13.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 -- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c105720 msgr2=0x7fb08c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c105720 0x7fb08c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 -- 172.21.15.129:0/3557826601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08c1a00b0 con 0x7fb08c102df0 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.986+0000 7fb090ca4700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c198450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb088006130 tx=0x7fb088004d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.987+0000 7fb081ffb700 1 -- 172.21.15.129:0/3557826601 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088021070 con 0x7fb08c102df0 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.987+0000 7fb081ffb700 1 -- 172.21.15.129:0/3557826601 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb088007b80 con 0x7fb08c102df0 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.987+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08c1a0340 con 0x7fb08c102df0 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.987+0000 7fb081ffb700 1 -- 172.21.15.129:0/3557826601 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0880052a0 con 0x7fb08c102df0 2024-03-16T09:36:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.987+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08c1a07f0 con 0x7fb08c102df0 2024-03-16T09:36:13.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.988+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb08c068f90 con 0x7fb08c102df0 2024-03-16T09:36:13.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.989+0000 7fb081ffb700 1 -- 172.21.15.129:0/3557826601 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb088024070 con 0x7fb08c102df0 2024-03-16T09:36:13.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.989+0000 7fb081ffb700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0740a2890 0x7fb0740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:13.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.989+0000 7fb081ffb700 1 -- 172.21.15.129:0/3557826601 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7fb08800f020 con 0x7fb08c102df0 2024-03-16T09:36:13.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.990+0000 7fb083fff700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0740a2890 0x7fb0740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:36:13.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.990+0000 7fb083fff700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0740a2890 0x7fb0740a4d50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009c60 tx=0x7fb07c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:36:13.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:13.994+0000 7fb081ffb700 1 -- 172.21.15.129:0/3557826601 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb0880d3070 con 0x7fb08c102df0 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] boot 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:14.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:14.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:14.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:14.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:14.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:14.197+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb08c0627d0 con 0x7fb0740a2890 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: osd.4 [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] boot 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:14.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:14.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:14.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:14 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi119 2024-03-16T09:36:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:14 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi119 to 8544M 2024-03-16T09:36:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:14 smithi129 ceph-mon[35841]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:36:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:36:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:15.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:14 smithi129 ceph-mon[35841]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T09:36:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:14 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi119 2024-03-16T09:36:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:14 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi119 to 8544M 2024-03-16T09:36:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:14 smithi119 ceph-mon[35632]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:36:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:36:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:14 smithi119 ceph-mon[35632]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T09:36:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:14 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi119 2024-03-16T09:36:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:14 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi119 to 8544M 2024-03-16T09:36:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:14 smithi077 ceph-mon[32171]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-16T09:36:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:36:15.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:36:15.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:15.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:14 smithi077 ceph-mon[32171]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T09:36:15.930 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:15 smithi129 ceph-mon[35841]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:36:16.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:15 smithi119 ceph-mon[35632]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:36:16.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:15 smithi077 ceph-mon[32171]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:36:17.133 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:16 smithi129 ceph-mon[35841]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:17.133 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:16 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/1539317051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]: dispatch 2024-03-16T09:36:17.133 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:16 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]: dispatch 2024-03-16T09:36:17.133 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:16 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]': finished 2024-03-16T09:36:17.133 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:16 smithi129 ceph-mon[35841]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T09:36:17.133 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:17.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:16 smithi119 ceph-mon[35632]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:16 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/1539317051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]: dispatch 2024-03-16T09:36:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:16 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]: dispatch 2024-03-16T09:36:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:16 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]': finished 2024-03-16T09:36:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:16 smithi119 ceph-mon[35632]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T09:36:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:16 smithi077 ceph-mon[32171]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:16 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/1539317051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]: dispatch 2024-03-16T09:36:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:16 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]: dispatch 2024-03-16T09:36:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:16 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0413402-bee8-4802-8b4a-d3a0a48049a1"}]': finished 2024-03-16T09:36:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:16 smithi077 ceph-mon[32171]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T09:36:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:18.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:17 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/2870245141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:36:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:17 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/2870245141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:36:18.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:17 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/2870245141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:36:19.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:18 smithi119 ceph-mon[35632]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:18 smithi077 ceph-mon[32171]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:19.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:18 smithi129 ceph-mon[35841]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:21.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:20 smithi129 ceph-mon[35841]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:21.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:20 smithi119 ceph-mon[35632]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:20 smithi077 ceph-mon[32171]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:23.015 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:22 smithi129 ceph-mon[35841]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:23.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:22 smithi119 ceph-mon[35632]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:23.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:22 smithi077 ceph-mon[32171]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:25.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:24 smithi129 ceph-mon[35841]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T09:36:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:25.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:24 smithi119 ceph-mon[35632]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:25.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T09:36:25.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:25.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:24 smithi077 ceph-mon[32171]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:25.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T09:36:25.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:26.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:25 smithi129 ceph-mon[35841]: Deploying daemon osd.5 on smithi129 2024-03-16T09:36:26.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:25 smithi119 ceph-mon[35632]: Deploying daemon osd.5 on smithi129 2024-03-16T09:36:26.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:25 smithi077 ceph-mon[32171]: Deploying daemon osd.5 on smithi129 2024-03-16T09:36:27.136 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:26 smithi129 ceph-mon[35841]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:27.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:26 smithi119 ceph-mon[35632]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:27.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:26 smithi077 ceph-mon[32171]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:29.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:28 smithi119 ceph-mon[35632]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:29.212 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:28 smithi129 ceph-mon[35841]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:28 smithi077 ceph-mon[32171]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:30.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:30.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:30.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:30.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:30 smithi129 ceph-mon[35841]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:30.689 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:30 smithi119 ceph-mon[35632]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:30 smithi077 ceph-mon[32171]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:32.953 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:32 smithi129 ceph-mon[35841]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:32.954 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:32.954 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:32.954 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:32.954 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:32.954 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:33.019 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 5 on host 'smithi129' 2024-03-16T09:36:33.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.016+0000 7fb081ffb700 1 -- 172.21.15.129:0/3557826601 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb08c0627d0 con 0x7fb0740a2890 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0740a2890 msgr2=0x7fb0740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0740a2890 0x7fb0740a4d50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009c60 tx=0x7fb07c009260 comp rx=0 tx=0).stop 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 msgr2=0x7fb08c198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c198450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb088006130 tx=0x7fb088004d70 comp rx=0 tx=0).stop 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 shutdown_connections 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0740a2890 0x7fb0740a4d50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08c108050 0x7fb08c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c102df0 0x7fb08c198450 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 --2- 172.21.15.129:0/3557826601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c105720 0x7fb08c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 >> 172.21.15.129:0/3557826601 conn(0x7fb08c0fc7a0 msgr2=0x7fb08c103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:36:33.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 shutdown_connections 2024-03-16T09:36:33.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:36:33.018+0000 7fb092f08700 1 -- 172.21.15.129:0/3557826601 wait complete. 2024-03-16T09:36:33.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:32 smithi119 ceph-mon[35632]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:32 smithi077 ceph-mon[32171]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:33.636 DEBUG:teuthology.orchestra.run.smithi129:osd.5> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.5.service 2024-03-16T09:36:33.638 INFO:tasks.cephadm:Deploying osd.6 on smithi077 with /dev/vg_nvme/lv_2... 2024-03-16T09:36:33.639 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T09:36:33.886 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:36:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:33 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:33 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:33 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:33 smithi077 ceph-mon[32171]: from='osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T09:36:34.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:33 smithi077 ceph-mon[32171]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T09:36:34.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:33 smithi077 ceph-mon[32171]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:33 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:33 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:33 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:33 smithi129 ceph-mon[35841]: from='osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T09:36:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:33 smithi129 ceph-mon[35841]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T09:36:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:33 smithi129 ceph-mon[35841]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:34.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:33 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:34.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:33 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:34.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:33 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:34.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:33 smithi119 ceph-mon[35632]: from='osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T09:36:34.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:33 smithi119 ceph-mon[35632]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T09:36:34.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:33 smithi119 ceph-mon[35632]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:35.255 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:35 smithi077 ceph-mon[32171]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T09:36:35.255 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:35 smithi077 ceph-mon[32171]: from='osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:36:35.256 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:35 smithi077 ceph-mon[32171]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T09:36:35.256 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:35.256 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:35 smithi077 ceph-mon[32171]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:36:35.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:35 smithi129 ceph-mon[35841]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T09:36:35.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:35 smithi129 ceph-mon[35841]: from='osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:36:35.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:35 smithi129 ceph-mon[35841]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T09:36:35.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:35.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:35 smithi129 ceph-mon[35841]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:36:35.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:35 smithi119 ceph-mon[35632]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T09:36:35.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:35 smithi119 ceph-mon[35632]: from='osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:36:35.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:35 smithi119 ceph-mon[35632]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T09:36:35.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:35.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:35 smithi119 ceph-mon[35632]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:36:35.819 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T09:36:35.819 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T09:36:35.819 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T09:36:35.819 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T09:36:35.819 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113548 s, 923 MB/s 2024-03-16T09:36:35.819 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 2024-03-16T09:36:35.819 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T09:36:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-mon[35841]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:36:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-mon[35841]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T09:36:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-mon[35841]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-mon[35841]: from='osd.5 ' entity='osd.5' 2024-03-16T09:36:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-mon[35841]: osdmap e38: 6 total, 5 up, 6 in 2024-03-16T09:36:36.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:36.052 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:36:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:36:35.936+0000 7f18bd3b9700 -1 osd.5 0 waiting for initial osdmap 2024-03-16T09:36:36.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:36 smithi077 ceph-mon[32171]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:36:36.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:36 smithi077 ceph-mon[32171]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T09:36:36.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:36.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:36 smithi077 ceph-mon[32171]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:36.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:36 smithi077 ceph-mon[32171]: from='osd.5 ' entity='osd.5' 2024-03-16T09:36:36.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:36 smithi077 ceph-mon[32171]: osdmap e38: 6 total, 5 up, 6 in 2024-03-16T09:36:36.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:36.427 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:36:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:36:36.054+0000 7f18b51a4700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:36:36.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:36 smithi119 ceph-mon[35632]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:36:36.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:36 smithi119 ceph-mon[35632]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T09:36:36.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:36 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:36.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:36 smithi119 ceph-mon[35632]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T09:36:36.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:36 smithi119 ceph-mon[35632]: from='osd.5 ' entity='osd.5' 2024-03-16T09:36:36.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:36 smithi119 ceph-mon[35632]: osdmap e38: 6 total, 5 up, 6 in 2024-03-16T09:36:36.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:36 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:36.476 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-03-16T09:36:36.723 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:36:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:36:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:36:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi129 2024-03-16T09:36:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi129 to 8544M 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] boot 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T09:36:37.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi129 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi129 to 8544M 2024-03-16T09:36:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] boot 2024-03-16T09:36:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T09:36:37.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:37.487 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi129 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi129 to 8544M 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: osd.5 [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] boot 2024-03-16T09:36:37.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T09:36:37.489 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/543500016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 msgr2=0x7f5fb8102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/543500016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8102720 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80099c0 tx=0x7f5fa8009cd0 comp rx=0 tx=0).stop 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/543500016 shutdown_connections 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/543500016 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb8103ea0 0x7f5fb810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/543500016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8102720 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/543500016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fb8103500 0x7f5fb8103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/543500016 >> 172.21.15.77:0/543500016 conn(0x7f5fb80fd8e0 msgr2=0x7f5fb80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:36:38.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/543500016 shutdown_connections 2024-03-16T09:36:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/543500016 wait complete. 2024-03-16T09:36:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 Processor -- start 2024-03-16T09:36:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.300+0000 7f5fbe9b9700 1 -- start start 2024-03-16T09:36:38.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb8103500 0x7f5fb8198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fb8103ea0 0x7f5fb819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb810de70 con 0x7f5fb8102300 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb810dcf0 con 0x7f5fb8103500 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb810dff0 con 0x7f5fb8103ea0 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54724/0 (socket says 172.21.15.77:54724) 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 -- 172.21.15.77:0/1937540958 learned_addr learned my addr 172.21.15.77:0/1937540958 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb8103500 0x7f5fb8198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 -- 172.21.15.77:0/1937540958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fb8103ea0 msgr2=0x7f5fb819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fb8103ea0 0x7f5fb819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:38.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 -- 172.21.15.77:0/1937540958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb8103500 msgr2=0x7f5fb8198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb8103500 0x7f5fb8198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 -- 172.21.15.77:0/1937540958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb81a0260 con 0x7f5fb8102300 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb8103500 0x7f5fb8198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb7fff700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8198600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8003680 tx=0x7f5fa8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb57fa700 1 -- 172.21.15.77:0/1937540958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8027070 con 0x7f5fb8102300 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb57fa700 1 -- 172.21.15.77:0/1937540958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fa8022470 con 0x7f5fb8102300 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb81a04f0 con 0x7f5fb8102300 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fb57fa700 1 -- 172.21.15.77:0/1937540958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa800f6c0 con 0x7f5fb8102300 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.301+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb81a0a50 con 0x7f5fb8102300 2024-03-16T09:36:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.303+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb8068f90 con 0x7f5fb8102300 2024-03-16T09:36:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.303+0000 7f5fb57fa700 1 -- 172.21.15.77:0/1937540958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5fa802a070 con 0x7f5fb8102300 2024-03-16T09:36:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.304+0000 7f5fb57fa700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fa00a2890 0x7f5fa00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:36:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.304+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fa00a2890 0x7f5fa00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:36:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.304+0000 7f5fb57fa700 1 -- 172.21.15.77:0/1937540958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f5fa80ceb40 con 0x7f5fb8102300 2024-03-16T09:36:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.305+0000 7f5fb77fe700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fa00a2890 0x7f5fa00a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0099d0 tx=0x7f5fac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:36:38.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.309+0000 7f5fb57fa700 1 -- 172.21.15.77:0/1937540958 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fa8013070 con 0x7f5fb8102300 2024-03-16T09:36:38.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:38.515+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5fb8062710 con 0x7f5fa00a2890 2024-03-16T09:36:38.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:38 smithi119 ceph-mon[35632]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T09:36:38.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:38 smithi119 ceph-mon[35632]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:38.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:38 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:36:38.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:38 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:36:38.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:38 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:39.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:38 smithi077 ceph-mon[32171]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T09:36:39.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:38 smithi077 ceph-mon[32171]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:39.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:36:39.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:36:39.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:39.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:38 smithi129 ceph-mon[35841]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T09:36:39.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:38 smithi129 ceph-mon[35841]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:39.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:38 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:36:39.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:38 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:36:39.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:38 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:39.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:39 smithi077 ceph-mon[32171]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:36:39.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:39 smithi077 ceph-mon[32171]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T09:36:39.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:39 smithi119 ceph-mon[35632]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:36:39.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:39 smithi119 ceph-mon[35632]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T09:36:40.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:39 smithi129 ceph-mon[35841]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:36:40.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:39 smithi129 ceph-mon[35841]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T09:36:40.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:40 smithi077 ceph-mon[32171]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T09:36:40.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:40 smithi077 ceph-mon[32171]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-03-16T09:36:40.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:40 smithi119 ceph-mon[35632]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T09:36:40.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:40 smithi119 ceph-mon[35632]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-03-16T09:36:41.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:40 smithi129 ceph-mon[35841]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T09:36:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:40 smithi129 ceph-mon[35841]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-03-16T09:36:41.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:41 smithi077 ceph-mon[32171]: osdmap e43: 6 total, 6 up, 6 in 2024-03-16T09:36:41.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:41 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1591642794' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6700df2-9147-4ff5-93f7-105320d2a596"}]: dispatch 2024-03-16T09:36:41.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:41 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1591642794' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6700df2-9147-4ff5-93f7-105320d2a596"}]': finished 2024-03-16T09:36:41.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:41 smithi077 ceph-mon[32171]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T09:36:41.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:36:41.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:41 smithi119 ceph-mon[35632]: osdmap e43: 6 total, 6 up, 6 in 2024-03-16T09:36:41.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:41 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1591642794' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6700df2-9147-4ff5-93f7-105320d2a596"}]: dispatch 2024-03-16T09:36:41.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:41 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1591642794' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6700df2-9147-4ff5-93f7-105320d2a596"}]': finished 2024-03-16T09:36:41.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:41 smithi119 ceph-mon[35632]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T09:36:41.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:36:42.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:41 smithi129 ceph-mon[35841]: osdmap e43: 6 total, 6 up, 6 in 2024-03-16T09:36:42.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:41 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1591642794' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6700df2-9147-4ff5-93f7-105320d2a596"}]: dispatch 2024-03-16T09:36:42.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:41 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1591642794' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6700df2-9147-4ff5-93f7-105320d2a596"}]': finished 2024-03-16T09:36:42.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:41 smithi129 ceph-mon[35841]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T09:36:42.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:36:42.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:42 smithi119 ceph-mon[35632]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T09:36:42.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:42 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1168533520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:36:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:42 smithi077 ceph-mon[32171]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T09:36:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:42 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1168533520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:36:43.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:42 smithi129 ceph-mon[35841]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T09:36:43.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:42 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1168533520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:36:45.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:44 smithi129 ceph-mon[35841]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-16T09:36:45.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:44 smithi077 ceph-mon[32171]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-16T09:36:45.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:44 smithi119 ceph-mon[35632]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-16T09:36:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:46 smithi077 ceph-mon[32171]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-16T09:36:47.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:46 smithi129 ceph-mon[35841]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-16T09:36:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:46 smithi119 ceph-mon[35632]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-16T09:36:49.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:48 smithi077 ceph-mon[32171]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:49.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T09:36:49.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:49.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:48 smithi129 ceph-mon[35841]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:49.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T09:36:49.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:48 smithi119 ceph-mon[35632]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T09:36:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:36:49.969 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:49 smithi077 ceph-mon[32171]: Deploying daemon osd.6 on smithi077 2024-03-16T09:36:50.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:49 smithi129 ceph-mon[35841]: Deploying daemon osd.6 on smithi077 2024-03-16T09:36:50.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:49 smithi119 ceph-mon[35632]: Deploying daemon osd.6 on smithi077 2024-03-16T09:36:51.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:50 smithi119 ceph-mon[35632]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:50 smithi077 ceph-mon[32171]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:51.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:50 smithi129 ceph-mon[35841]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:53.091 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:52 smithi077 ceph-mon[32171]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:53.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:52 smithi119 ceph-mon[35632]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:52 smithi129 ceph-mon[35841]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:54 smithi119 ceph-mon[35632]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:54 smithi077 ceph-mon[32171]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:54 smithi129 ceph-mon[35841]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:36:56.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:56 smithi077 ceph-mon[32171]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:56 smithi119 ceph-mon[35632]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:57.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:56 smithi129 ceph-mon[35841]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:57.323 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 6 on host 'smithi077' 2024-03-16T09:36:57.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.319+0000 7f5fb57fa700 1 -- 172.21.15.77:0/1937540958 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5fb8062710 con 0x7f5fa00a2890 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fa00a2890 msgr2=0x7f5fa00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fa00a2890 0x7f5fa00a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0099d0 tx=0x7f5fac008040 comp rx=0 tx=0).stop 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 msgr2=0x7f5fb8198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8198600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8003680 tx=0x7f5fa8005e80 comp rx=0 tx=0).stop 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 shutdown_connections 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fa00a2890 0x7f5fa00a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fb8103ea0 0x7f5fb819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:57.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102300 0x7f5fb8198600 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 --2- 172.21.15.77:0/1937540958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb8103500 0x7f5fb8198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:36:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 >> 172.21.15.77:0/1937540958 conn(0x7f5fb80fd8e0 msgr2=0x7f5fb8106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:36:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 shutdown_connections 2024-03-16T09:36:57.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:36:57.322+0000 7f5fbe9b9700 1 -- 172.21.15.77:0/1937540958 wait complete. 2024-03-16T09:36:57.945 DEBUG:teuthology.orchestra.run.smithi077:osd.6> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.6.service 2024-03-16T09:36:57.947 INFO:tasks.cephadm:Deploying osd.7 on smithi119 with /dev/vg_nvme/lv_2... 2024-03-16T09:36:57.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T09:36:58.195 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:36:58.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:36:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:36:57.996+0000 7f8fd88bb7c0 -1 osd.6 0 log_to_monitors true 2024-03-16T09:36:58.563 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.563 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.563 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:58 smithi119 ceph-mon[35632]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:58.563 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.563 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:58 smithi119 ceph-mon[35632]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T09:36:58.563 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:36:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.604 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.605 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.605 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:58 smithi077 ceph-mon[32171]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:58.605 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.605 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:58 smithi077 ceph-mon[32171]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T09:36:58.605 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:36:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:58 smithi129 ceph-mon[35841]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:36:58.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:36:58.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:58 smithi129 ceph-mon[35841]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T09:36:58.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:36:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:00.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:00 smithi119 ceph-mon[35632]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T09:37:00.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:00 smithi119 ceph-mon[35632]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T09:37:00.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:00 smithi119 ceph-mon[35632]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:37:00.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:00 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:00.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:00 smithi119 ceph-mon[35632]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:37:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T09:37:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T09:37:00.243 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-16T09:37:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-16T09:37:00.243 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112401 s, 933 MB/s 2024-03-16T09:37:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-16T09:37:00.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:00 smithi077 ceph-mon[32171]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T09:37:00.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:00 smithi077 ceph-mon[32171]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T09:37:00.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:00 smithi077 ceph-mon[32171]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:37:00.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:00 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:00.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:00 smithi077 ceph-mon[32171]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:37:00.260 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:37:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:37:00.013+0000 7f8fcdf20700 -1 osd.6 0 waiting for initial osdmap 2024-03-16T09:37:00.260 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:37:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:37:00.022+0000 7f8fc9d13700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:37:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:00 smithi129 ceph-mon[35841]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T09:37:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:00 smithi129 ceph-mon[35841]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T09:37:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:00 smithi129 ceph-mon[35841]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:37:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:00 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:00 smithi129 ceph-mon[35841]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-16T09:37:00.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-03-16T09:37:01.073 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:37:01.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:01 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:37:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:01 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:37:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:01 smithi119 ceph-mon[35632]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:37:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:01 smithi119 ceph-mon[35632]: osdmap e46: 7 total, 6 up, 7 in 2024-03-16T09:37:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:01 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:01 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:01.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:01 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:37:01.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:01 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:37:01.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:01 smithi077 ceph-mon[32171]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:37:01.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:01 smithi077 ceph-mon[32171]: osdmap e46: 7 total, 6 up, 7 in 2024-03-16T09:37:01.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:01.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:01 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:01 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:37:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:01 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:37:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:01 smithi129 ceph-mon[35841]: from='osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:37:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:01 smithi129 ceph-mon[35841]: osdmap e46: 7 total, 6 up, 7 in 2024-03-16T09:37:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:01 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:01 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] boot 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi077 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi077 to 5696M 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.150 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:02.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:02.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:02.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:02.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:02.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.151 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:02 smithi119 ceph-mon[35632]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] boot 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi077 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi077 to 5696M 2024-03-16T09:37:02.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:02 smithi129 ceph-mon[35841]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: osd.6 [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] boot 2024-03-16T09:37:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T09:37:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T09:37:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi077 2024-03-16T09:37:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi077 to 5696M 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:02.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:02 smithi077 ceph-mon[32171]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:02.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 -- 172.21.15.119:0/247921640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 msgr2=0x7f89a81026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:02.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 --2- 172.21.15.119:0/247921640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a81026f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89980099c0 tx=0x7f8998009cd0 comp rx=0 tx=0).stop 2024-03-16T09:37:02.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 -- 172.21.15.119:0/247921640 shutdown_connections 2024-03-16T09:37:02.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 --2- 172.21.15.119:0/247921640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a81026f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 --2- 172.21.15.119:0/247921640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a8103e70 0x7f89a810a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 --2- 172.21.15.119:0/247921640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a81034d0 0x7f89a8103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 -- 172.21.15.119:0/247921640 >> 172.21.15.119:0/247921640 conn(0x7f89a80fd8b0 msgr2=0x7f89a80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 -- 172.21.15.119:0/247921640 shutdown_connections 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.662+0000 7f89afe3c700 1 -- 172.21.15.119:0/247921640 wait complete. 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 Processor -- start 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 -- start start 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a81034d0 0x7f89a8198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a8103e70 0x7f89a819f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89a810dd40 con 0x7f89a8103e70 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f89a810dbc0 con 0x7f89a81022d0 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89afe3c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89a810dec0 con 0x7f89a81034d0 2024-03-16T09:37:02.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89adbd8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89adbd8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a8198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50764/0 (socket says 172.21.15.119:50764) 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89adbd8700 1 -- 172.21.15.119:0/199386407 learned_addr learned my addr 172.21.15.119:0/199386407 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89ae3d9700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a8103e70 0x7f89a819f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.663+0000 7f89ad3d7700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a81034d0 0x7f89a8198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89adbd8700 1 -- 172.21.15.119:0/199386407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a81034d0 msgr2=0x7f89a8198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89adbd8700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a81034d0 0x7f89a8198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89adbd8700 1 -- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a8103e70 msgr2=0x7f89a819f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89adbd8700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a8103e70 0x7f89a819f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89adbd8700 1 -- 172.21.15.119:0/199386407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a81a0080 con 0x7f89a81022d0 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89ad3d7700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a81034d0 0x7f89a8198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89adbd8700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a8198420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f899800c010 tx=0x7f8998009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f899effd700 1 -- 172.21.15.119:0/199386407 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998027070 con 0x7f89a81022d0 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f899effd700 1 -- 172.21.15.119:0/199386407 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89980040d0 con 0x7f89a81022d0 2024-03-16T09:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a81a0310 con 0x7f89a81022d0 2024-03-16T09:37:02.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f899effd700 1 -- 172.21.15.119:0/199386407 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998022d40 con 0x7f89a81022d0 2024-03-16T09:37:02.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.664+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a81a0840 con 0x7f89a81022d0 2024-03-16T09:37:02.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.665+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a8068f90 con 0x7f89a81022d0 2024-03-16T09:37:02.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.666+0000 7f899effd700 1 -- 172.21.15.119:0/199386407 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f899802a070 con 0x7f89a81022d0 2024-03-16T09:37:02.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.667+0000 7f899effd700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f89940a2990 0x7f89940a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:02.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.667+0000 7f899effd700 1 -- 172.21.15.119:0/199386407 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f89980cf1e0 con 0x7f89a81022d0 2024-03-16T09:37:02.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.667+0000 7f89ad3d7700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f89940a2990 0x7f89940a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:02.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.667+0000 7f89ad3d7700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f89940a2990 0x7f89940a4e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89a4005fd0 tx=0x7f89a4005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:37:02.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.670+0000 7f899effd700 1 -- 172.21.15.119:0/199386407 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8998031070 con 0x7f89a81022d0 2024-03-16T09:37:02.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:02.870+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f89a8062710 con 0x7f89940a2990 2024-03-16T09:37:03.722 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:03 smithi119 ceph-mon[35632]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T09:37:03.722 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:03 smithi119 ceph-mon[35632]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:03.722 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:03 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:03.723 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:03 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:03.723 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:03 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:03.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:03 smithi077 ceph-mon[32171]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T09:37:03.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:03 smithi077 ceph-mon[32171]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:03.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:03 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:03.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:03 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:03.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:03 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:03.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:03 smithi129 ceph-mon[35841]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T09:37:03.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:03 smithi129 ceph-mon[35841]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:03.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:03 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:03.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:03 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:03.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:03 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:04.752 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:04 smithi119 ceph-mon[35632]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:04.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:04 smithi077 ceph-mon[32171]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:04.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:04 smithi129 ceph-mon[35841]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:05.912 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:05 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/814389140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]: dispatch 2024-03-16T09:37:05.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:05 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]: dispatch 2024-03-16T09:37:05.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:05 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]': finished 2024-03-16T09:37:05.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:05 smithi119 ceph-mon[35632]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T09:37:05.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:05 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:05 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/814389140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]: dispatch 2024-03-16T09:37:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:05 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]: dispatch 2024-03-16T09:37:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:05 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]': finished 2024-03-16T09:37:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:05 smithi077 ceph-mon[32171]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T09:37:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:05 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:06.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:05 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/814389140' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]: dispatch 2024-03-16T09:37:06.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:05 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]: dispatch 2024-03-16T09:37:06.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:05 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f717036c-9f74-49f7-9e6a-5a220593a044"}]': finished 2024-03-16T09:37:06.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:05 smithi129 ceph-mon[35841]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T09:37:06.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:05 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:06 smithi119 ceph-mon[35632]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:06 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/477194538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:06 smithi077 ceph-mon[32171]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:06 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/477194538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:07.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:06 smithi129 ceph-mon[35841]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:07.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:06 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/477194538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:09.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:08 smithi129 ceph-mon[35841]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:08 smithi119 ceph-mon[35632]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:09.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:08 smithi077 ceph-mon[32171]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:10.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:10 smithi119 ceph-mon[35632]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:11.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:10 smithi129 ceph-mon[35841]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:10 smithi077 ceph-mon[32171]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:12.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:12 smithi119 ceph-mon[35632]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:13.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:12 smithi129 ceph-mon[35841]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:13.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:12 smithi077 ceph-mon[32171]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:14.188 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T09:37:14.188 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:13 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T09:37:14.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:13 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:14.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T09:37:14.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:13 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:15.158 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:14 smithi119 ceph-mon[35632]: Deploying daemon osd.7 on smithi119 2024-03-16T09:37:15.158 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:14 smithi119 ceph-mon[35632]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:14 smithi077 ceph-mon[32171]: Deploying daemon osd.7 on smithi119 2024-03-16T09:37:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:14 smithi077 ceph-mon[32171]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:14 smithi129 ceph-mon[35841]: Deploying daemon osd.7 on smithi119 2024-03-16T09:37:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:14 smithi129 ceph-mon[35841]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:16 smithi119 ceph-mon[35632]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:16 smithi077 ceph-mon[32171]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:16 smithi129 ceph-mon[35841]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:19.145 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:18 smithi119 ceph-mon[35632]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:19.146 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:19.146 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:19.146 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:18 smithi077 ceph-mon[32171]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:18 smithi129 ceph-mon[35841]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:20.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:20 smithi119 ceph-mon[35632]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:20 smithi077 ceph-mon[32171]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:21.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:20 smithi129 ceph-mon[35841]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:22.187 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 7 on host 'smithi119' 2024-03-16T09:37:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.183+0000 7f899effd700 1 -- 172.21.15.119:0/199386407 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f89a8062710 con 0x7f89940a2990 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f89940a2990 msgr2=0x7f89940a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f89940a2990 0x7f89940a4e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89a4005fd0 tx=0x7f89a4005dc0 comp rx=0 tx=0).stop 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 msgr2=0x7f89a8198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a8198420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f899800c010 tx=0x7f8998009580 comp rx=0 tx=0).stop 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 shutdown_connections 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f89940a2990 0x7f89940a4e50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a81022d0 0x7f89a8198420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89a8103e70 0x7f89a819f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 --2- 172.21.15.119:0/199386407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a81034d0 0x7f89a8198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 >> 172.21.15.119:0/199386407 conn(0x7f89a80fd8b0 msgr2=0x7f89a8106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 shutdown_connections 2024-03-16T09:37:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:37:22.186+0000 7f89afe3c700 1 -- 172.21.15.119:0/199386407 wait complete. 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:22 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:22.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:22.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:22.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:22.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:22 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:22.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:22 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:22.812 DEBUG:teuthology.orchestra.run.smithi119:osd.7> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.7.service 2024-03-16T09:37:22.815 INFO:tasks.cephadm:Deploying osd.8 on smithi129 with /dev/vg_nvme/lv_2... 2024-03-16T09:37:22.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T09:37:23.060 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:37:23.403 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:23 smithi129 ceph-mon[35841]: from='osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T09:37:23.404 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:23 smithi129 ceph-mon[35841]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T09:37:23.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:23 smithi119 ceph-mon[35632]: from='osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T09:37:23.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:23 smithi119 ceph-mon[35632]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T09:37:23.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:23 smithi077 ceph-mon[32171]: from='osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T09:37:23.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:23 smithi077 ceph-mon[32171]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T09:37:24.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-mon[35632]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T09:37:24.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-mon[35632]: from='osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:37:24.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-mon[35632]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T09:37:24.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:24.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-mon[35632]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:37:24.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-mon[35632]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:24.462 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:24 smithi129 ceph-mon[35841]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T09:37:24.462 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:24 smithi129 ceph-mon[35841]: from='osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:37:24.462 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:24 smithi129 ceph-mon[35841]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T09:37:24.462 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:24.462 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:24 smithi129 ceph-mon[35841]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:37:24.462 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:24 smithi129 ceph-mon[35841]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:24.690 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:37:24.361+0000 7fed25824700 -1 osd.7 0 waiting for initial osdmap 2024-03-16T09:37:24.690 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:37:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:37:24.370+0000 7fed21617700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:37:24.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:24 smithi077 ceph-mon[32171]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T09:37:24.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:24 smithi077 ceph-mon[32171]: from='osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:37:24.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:24 smithi077 ceph-mon[32171]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T09:37:24.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:24.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:24 smithi077 ceph-mon[32171]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:37:24.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:24 smithi077 ceph-mon[32171]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:25.036 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T09:37:25.036 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T09:37:25.036 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-16T09:37:25.036 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-16T09:37:25.036 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111597 s, 940 MB/s 2024-03-16T09:37:25.036 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-16T09:37:25.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:25 smithi129 ceph-mon[35841]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:37:25.630 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:25 smithi129 ceph-mon[35841]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T09:37:25.631 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:25 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:25.631 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:25 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:25.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2024-03-16T09:37:25.675 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:25 smithi119 ceph-mon[35632]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:37:25.675 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:25 smithi119 ceph-mon[35632]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T09:37:25.675 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:25 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:25.675 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:25 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:25.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:25 smithi077 ceph-mon[32171]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:37:25.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:25 smithi077 ceph-mon[32171]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T09:37:25.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:25 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:25.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:25 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] boot 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi119 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.686 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi119 to 5696M 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:26.687 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:37:26.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:37:26.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:26.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] boot 2024-03-16T09:37:26.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T09:37:26.692 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi119 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi119 to 5696M 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:26.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:37:26.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:37:26.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:26.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: osd.7 [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] boot 2024-03-16T09:37:26.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi119 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi119 to 5696M 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:26.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:27.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 -- 172.21.15.129:0/3224527480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0102300 msgr2=0x7fdcd0102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/3224527480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0102300 0x7fdcd0102720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009990 tx=0x7fdcc0009ca0 comp rx=0 tx=0).stop 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 -- 172.21.15.129:0/3224527480 shutdown_connections 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/3224527480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd0103ea0 0x7fdcd010a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/3224527480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0102300 0x7fdcd0102720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/3224527480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcd0103500 0x7fdcd0103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 -- 172.21.15.129:0/3224527480 >> 172.21.15.129:0/3224527480 conn(0x7fdcd00fd8e0 msgr2=0x7fdcd00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 -- 172.21.15.129:0/3224527480 shutdown_connections 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 -- 172.21.15.129:0/3224527480 wait complete. 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 Processor -- start 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.467+0000 7fdcd5b98700 1 -- start start 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcd5b98700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd0102300 0x7fdcd01984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcd5b98700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcd0103500 0x7fdcd0198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcd5b98700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0103ea0 0x7fdcd019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcd5b98700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd010ddf0 con 0x7fdcd0102300 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcd5b98700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd010dc70 con 0x7fdcd0103500 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcd5b98700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd010df70 con 0x7fdcd0103ea0 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0103ea0 0x7fdcd019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0103ea0 0x7fdcd019faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57032/0 (socket says 172.21.15.129:57032) 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 -- 172.21.15.129:0/874661470 learned_addr learned my addr 172.21.15.129:0/874661470 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccf7fe700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd0102300 0x7fdcd01984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcc6ffd700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcd0103500 0x7fdcd0198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 -- 172.21.15.129:0/874661470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcd0103500 msgr2=0x7fdcd0198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcd0103500 0x7fdcd0198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 -- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd0102300 msgr2=0x7fdcd01984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd0102300 0x7fdcd01984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:27.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 -- 172.21.15.129:0/874661470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd01a0140 con 0x7fdcd0103ea0 2024-03-16T09:37:27.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccffff700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0103ea0 0x7fdcd019faa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800b9a0 tx=0x7fdcc800bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:37:27.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdccd7fa700 1 -- 172.21.15.129:0/874661470 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800d850 con 0x7fdcd0103ea0 2024-03-16T09:37:27.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.468+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd01a0430 con 0x7fdcd0103ea0 2024-03-16T09:37:27.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.469+0000 7fdccd7fa700 1 -- 172.21.15.129:0/874661470 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc800e420 con 0x7fdcd0103ea0 2024-03-16T09:37:27.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.469+0000 7fdccd7fa700 1 -- 172.21.15.129:0/874661470 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8017660 con 0x7fdcd0103ea0 2024-03-16T09:37:27.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.469+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd01a09f0 con 0x7fdcd0103ea0 2024-03-16T09:37:27.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.470+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd0068f90 con 0x7fdcd0103ea0 2024-03-16T09:37:27.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.472+0000 7fdccd7fa700 1 -- 172.21.15.129:0/874661470 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdcc800e5d0 con 0x7fdcd0103ea0 2024-03-16T09:37:27.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.472+0000 7fdccd7fa700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:27.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.472+0000 7fdccd7fa700 1 -- 172.21.15.129:0/874661470 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7fdcc80cdc50 con 0x7fdcd0103ea0 2024-03-16T09:37:27.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.472+0000 7fdccf7fe700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:27.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.473+0000 7fdccf7fe700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00037f0 tx=0x7fdcc0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:37:27.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.476+0000 7fdccd7fa700 1 -- 172.21.15.129:0/874661470 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdcc8013030 con 0x7fdcd0103ea0 2024-03-16T09:37:27.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:27 smithi129 ceph-mon[35841]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T09:37:27.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:27.681+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fdcd0062710 con 0x7fdcbc0a2890 2024-03-16T09:37:27.689 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:27 smithi119 ceph-mon[35632]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T09:37:27.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:27 smithi077 ceph-mon[32171]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T09:37:28.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:28 smithi129 ceph-mon[35841]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T09:37:28.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:28 smithi129 ceph-mon[35841]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:28.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:28.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:28.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:28.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:28 smithi129 ceph-mon[35841]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:28 smithi119 ceph-mon[35632]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T09:37:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:28 smithi119 ceph-mon[35632]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:28.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:28 smithi119 ceph-mon[35632]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:28 smithi077 ceph-mon[32171]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T09:37:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:28 smithi077 ceph-mon[32171]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:28.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:28 smithi077 ceph-mon[32171]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:29.500 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:29 smithi129 ceph-mon[35841]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T09:37:29.689 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:29 smithi119 ceph-mon[35632]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T09:37:29.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:29 smithi077 ceph-mon[32171]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T09:37:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:30 smithi119 ceph-mon[35632]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T09:37:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:30 smithi119 ceph-mon[35632]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:30 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/3684395031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]: dispatch 2024-03-16T09:37:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:30 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]: dispatch 2024-03-16T09:37:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:30 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]': finished 2024-03-16T09:37:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:30 smithi119 ceph-mon[35632]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T09:37:30.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:30.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:30 smithi129 ceph-mon[35841]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T09:37:30.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:30 smithi129 ceph-mon[35841]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:30.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:30 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/3684395031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]: dispatch 2024-03-16T09:37:30.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:30 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]: dispatch 2024-03-16T09:37:30.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:30 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]': finished 2024-03-16T09:37:30.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:30 smithi129 ceph-mon[35841]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T09:37:30.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:30 smithi077 ceph-mon[32171]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T09:37:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:30 smithi077 ceph-mon[32171]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:30 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/3684395031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]: dispatch 2024-03-16T09:37:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:30 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]: dispatch 2024-03-16T09:37:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:30 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02a6c20f-3922-494c-a45a-84173f7c41c4"}]': finished 2024-03-16T09:37:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:30 smithi077 ceph-mon[32171]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T09:37:30.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:31.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:31 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/2393170576' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:31.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:31 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/2393170576' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:31.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:31 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/2393170576' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:32.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:32 smithi119 ceph-mon[35632]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:32.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:32 smithi077 ceph-mon[32171]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:32.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:32 smithi129 ceph-mon[35841]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:34.928 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:34 smithi129 ceph-mon[35841]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:35.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:34 smithi119 ceph-mon[35632]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:34 smithi077 ceph-mon[32171]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:37.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:36 smithi129 ceph-mon[35841]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:37.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:36 smithi119 ceph-mon[35632]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:36 smithi077 ceph-mon[32171]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:38.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T09:37:38.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T09:37:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T09:37:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:39.163 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:38 smithi129 ceph-mon[35841]: Deploying daemon osd.8 on smithi129 2024-03-16T09:37:39.163 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:38 smithi129 ceph-mon[35841]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:39.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:38 smithi119 ceph-mon[35632]: Deploying daemon osd.8 on smithi129 2024-03-16T09:37:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:38 smithi119 ceph-mon[35632]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:38 smithi077 ceph-mon[32171]: Deploying daemon osd.8 on smithi129 2024-03-16T09:37:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:38 smithi077 ceph-mon[32171]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:40 smithi119 ceph-mon[35632]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:41.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:40 smithi077 ceph-mon[32171]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:41.272 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:40 smithi129 ceph-mon[35841]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:42.905 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:42 smithi129 ceph-mon[35841]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:43.182 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:42 smithi077 ceph-mon[32171]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:43.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:42 smithi119 ceph-mon[35632]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:44.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:44 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:44.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:44 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:44.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:44 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:44.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:44 smithi077 ceph-mon[32171]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:44.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:44 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:44.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:44 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:44.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:44 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:44.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:44 smithi119 ceph-mon[35632]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:44.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:44 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:44.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:44 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:44.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:44 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:44.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:44 smithi129 ceph-mon[35841]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:46.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.897+0000 7fdccd7fa700 1 -- 172.21.15.129:0/874661470 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdcd0062710 con 0x7fdcbc0a2890 2024-03-16T09:37:46.900 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 8 on host 'smithi129' 2024-03-16T09:37:46.900 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:46 smithi129 ceph-mon[35841]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:46.901 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:46.901 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:46.901 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:46.901 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:46.901 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:46.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.898+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdcbc0a2890 msgr2=0x7fdcbc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:46.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.898+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00037f0 tx=0x7fdcc0005fd0 comp rx=0 tx=0).stop 2024-03-16T09:37:46.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.898+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0103ea0 msgr2=0x7fdcd019faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.898+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0103ea0 0x7fdcd019faa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800b9a0 tx=0x7fdcc800bcb0 comp rx=0 tx=0).stop 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 shutdown_connections 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd0103ea0 0x7fdcd019faa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd0102300 0x7fdcd01984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 --2- 172.21.15.129:0/874661470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcd0103500 0x7fdcd0198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 >> 172.21.15.129:0/874661470 conn(0x7fdcd00fd8e0 msgr2=0x7fdcd01068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 shutdown_connections 2024-03-16T09:37:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:37:46.899+0000 7fdcd5b98700 1 -- 172.21.15.129:0/874661470 wait complete. 2024-03-16T09:37:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:46 smithi119 ceph-mon[35632]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:46 smithi077 ceph-mon[32171]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:47.517 DEBUG:teuthology.orchestra.run.smithi129:osd.8> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.8.service 2024-03-16T09:37:47.519 INFO:tasks.cephadm:Deploying osd.9 on smithi077 with /dev/vg_nvme/lv_1... 2024-03-16T09:37:47.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T09:37:47.767 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:37:48.123 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:47 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:48.123 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:47 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:48.123 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:47 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:48.123 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:47 smithi077 ceph-mon[32171]: from='osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T09:37:48.123 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:47 smithi077 ceph-mon[32171]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T09:37:48.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:47 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:48.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:47 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:48.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:47 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:48.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:47 smithi119 ceph-mon[35632]: from='osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T09:37:48.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:47 smithi119 ceph-mon[35632]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T09:37:48.298 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:47 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:48.298 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:47 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:48.298 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:47 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:37:48.298 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:47 smithi129 ceph-mon[35841]: from='osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T09:37:48.298 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:47 smithi129 ceph-mon[35841]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T09:37:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:48 smithi119 ceph-mon[35632]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:48 smithi119 ceph-mon[35632]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T09:37:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:48 smithi119 ceph-mon[35632]: from='osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:37:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:48 smithi119 ceph-mon[35632]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T09:37:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:48 smithi119 ceph-mon[35632]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:37:49.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:48 smithi077 ceph-mon[32171]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:49.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:48 smithi077 ceph-mon[32171]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T09:37:49.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:48 smithi077 ceph-mon[32171]: from='osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:37:49.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:48 smithi077 ceph-mon[32171]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T09:37:49.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:49.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:48 smithi077 ceph-mon[32171]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:37:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-mon[35841]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-mon[35841]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T09:37:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-mon[35841]: from='osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:37:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-mon[35841]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T09:37:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-mon[35841]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:37:49.301 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:37:48.889+0000 7f0004308700 -1 osd.8 0 waiting for initial osdmap 2024-03-16T09:37:49.301 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:37:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:37:48.901+0000 7f00000fb700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:37:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T09:37:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T09:37:49.753 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T09:37:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T09:37:49.753 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113471 s, 924 MB/s 2024-03-16T09:37:49.753 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T09:37:50.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:49 smithi129 ceph-mon[35841]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:37:50.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:49 smithi129 ceph-mon[35841]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T09:37:50.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:49 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:50.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:49 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:50.013 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:49 smithi129 ceph-mon[35841]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:50.139 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:49 smithi077 ceph-mon[32171]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:37:50.139 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:49 smithi077 ceph-mon[32171]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T09:37:50.139 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:49 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:50.139 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:49 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:50.140 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:49 smithi077 ceph-mon[32171]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:50.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:49 smithi119 ceph-mon[35632]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:37:50.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:49 smithi119 ceph-mon[35632]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T09:37:50.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:49 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:50.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:49 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:50.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:49 smithi119 ceph-mon[35632]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-16T09:37:50.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-03-16T09:37:50.720 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] boot 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi129 2024-03-16T09:37:51.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi129 to 5696M 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:51.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] boot 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi129 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi129 to 5696M 2024-03-16T09:37:51.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:51.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:51.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: osd.8 [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] boot 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi129 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi129 to 5696M 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:37:51.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:37:51.987 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:51 smithi077 ceph-mon[32171]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T09:37:51.988 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:51 smithi077 ceph-mon[32171]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:52.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:51 smithi129 ceph-mon[35841]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T09:37:52.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:51 smithi129 ceph-mon[35841]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:52.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:51 smithi119 ceph-mon[35632]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T09:37:52.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:51 smithi119 ceph-mon[35632]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.225+0000 7f7b16168700 1 -- 172.21.15.77:0/1912500688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 msgr2=0x7f7b10107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.225+0000 7f7b16168700 1 --2- 172.21.15.77:0/1912500688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10107ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009990 tx=0x7f7b00009ca0 comp rx=0 tx=0).stop 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.225+0000 7f7b16168700 1 -- 172.21.15.77:0/1912500688 shutdown_connections 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.225+0000 7f7b16168700 1 --2- 172.21.15.77:0/1912500688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b10102d90 0x7f7b10105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.225+0000 7f7b16168700 1 --2- 172.21.15.77:0/1912500688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b10107ff0 0x7f7b1010a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.225+0000 7f7b16168700 1 --2- 172.21.15.77:0/1912500688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10107ab0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 -- 172.21.15.77:0/1912500688 >> 172.21.15.77:0/1912500688 conn(0x7f7b100fc7c0 msgr2=0x7f7b100febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 -- 172.21.15.77:0/1912500688 shutdown_connections 2024-03-16T09:37:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 -- 172.21.15.77:0/1912500688 wait complete. 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 Processor -- start 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 -- start start 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b10102d90 0x7f7b101026a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b10107ff0 0x7f7b10101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.226+0000 7f7b16168700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1010de00 con 0x7f7b101056c0 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b16168700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1010dc80 con 0x7f7b10107ff0 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b16168700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1010df80 con 0x7f7b10102d90 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54030/0 (socket says 172.21.15.77:54030) 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 -- 172.21.15.77:0/3020338070 learned_addr learned my addr 172.21.15.77:0/3020338070 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0f7fe700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b10102d90 0x7f7b101026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0ffff700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b10107ff0 0x7f7b10101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 -- 172.21.15.77:0/3020338070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b10102d90 msgr2=0x7f7b101026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b10102d90 0x7f7b101026a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 -- 172.21.15.77:0/3020338070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b10107ff0 msgr2=0x7f7b10101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b10107ff0 0x7f7b10101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 -- 172.21.15.77:0/3020338070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b10101b50 con 0x7f7b101056c0 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0effd700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10100d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000b5c0 tx=0x7f7b000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0cff9700 1 -- 172.21.15.77:0/3020338070 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00027070 con 0x7f7b101056c0 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0cff9700 1 -- 172.21.15.77:0/3020338070 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b00022470 con 0x7f7b101056c0 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b101a25b0 con 0x7f7b101056c0 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b0cff9700 1 -- 172.21.15.77:0/3020338070 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000f6c0 con 0x7f7b101056c0 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.227+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b101a2a60 con 0x7f7b101056c0 2024-03-16T09:37:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.228+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b10068f90 con 0x7f7b101056c0 2024-03-16T09:37:52.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.230+0000 7f7b0cff9700 1 -- 172.21.15.77:0/3020338070 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7b0002a070 con 0x7f7b101056c0 2024-03-16T09:37:52.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.230+0000 7f7b0cff9700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7afc0a2890 0x7f7afc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:37:52.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.231+0000 7f7b0f7fe700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7afc0a2890 0x7f7afc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:37:52.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.231+0000 7f7b0cff9700 1 -- 172.21.15.77:0/3020338070 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5858+0+0 (secure 0 0 0) 0x7f7b000cf850 con 0x7f7b101056c0 2024-03-16T09:37:52.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.231+0000 7f7b0f7fe700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7afc0a2890 0x7f7afc0a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7af8005950 tx=0x7f7af80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:37:52.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.234+0000 7f7b0cff9700 1 -- 172.21.15.77:0/3020338070 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b00031070 con 0x7f7b101056c0 2024-03-16T09:37:52.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:37:52.440+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7b10062710 con 0x7f7afc0a2890 2024-03-16T09:37:53.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:52 smithi077 ceph-mon[32171]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T09:37:53.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:52 smithi077 ceph-mon[32171]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:53.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:53.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:53.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:53.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:52 smithi077 ceph-mon[32171]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T09:37:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:52 smithi119 ceph-mon[35632]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T09:37:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:52 smithi119 ceph-mon[35632]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:52 smithi119 ceph-mon[35632]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T09:37:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:52 smithi129 ceph-mon[35841]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T09:37:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:52 smithi129 ceph-mon[35841]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:37:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:37:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:37:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:37:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:52 smithi129 ceph-mon[35841]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T09:37:54.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:54 smithi119 ceph-mon[35632]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T09:37:54.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:54 smithi119 ceph-mon[35632]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:55.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:54 smithi077 ceph-mon[32171]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T09:37:55.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:54 smithi077 ceph-mon[32171]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:55.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:54 smithi129 ceph-mon[35841]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T09:37:55.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:54 smithi129 ceph-mon[35841]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:55.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1831925492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e592022-204a-4db0-b470-63978907639a"}]: dispatch 2024-03-16T09:37:55.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1831925492' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e592022-204a-4db0-b470-63978907639a"}]': finished 2024-03-16T09:37:55.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:55 smithi119 ceph-mon[35632]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T09:37:55.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:37:56.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1831925492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e592022-204a-4db0-b470-63978907639a"}]: dispatch 2024-03-16T09:37:56.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1831925492' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e592022-204a-4db0-b470-63978907639a"}]': finished 2024-03-16T09:37:56.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:55 smithi077 ceph-mon[32171]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T09:37:56.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:37:56.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1831925492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e592022-204a-4db0-b470-63978907639a"}]: dispatch 2024-03-16T09:37:56.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1831925492' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e592022-204a-4db0-b470-63978907639a"}]': finished 2024-03-16T09:37:56.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:55 smithi129 ceph-mon[35841]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T09:37:56.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:37:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:56 smithi077 ceph-mon[32171]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:57.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:56 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3579394293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:57.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:56 smithi129 ceph-mon[35841]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:57.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:56 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3579394293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:56 smithi119 ceph-mon[35632]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:37:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:56 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3579394293' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:37:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:37:58 smithi119 ceph-mon[35632]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-16T09:37:59.228 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:37:58 smithi077 ceph-mon[32171]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-16T09:37:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:37:58 smithi129 ceph-mon[35841]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-16T09:38:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:00 smithi077 ceph-mon[32171]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:38:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:00 smithi119 ceph-mon[35632]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:38:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:00 smithi129 ceph-mon[35841]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:38:02.900 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:02 smithi077 ceph-mon[32171]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-16T09:38:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:02 smithi119 ceph-mon[35632]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-16T09:38:03.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:02 smithi129 ceph-mon[35841]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-16T09:38:04.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:03 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T09:38:04.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:03 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:04.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:03 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T09:38:04.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:03 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:04.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:03 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T09:38:04.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:03 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:04 smithi077 ceph-mon[32171]: Deploying daemon osd.9 on smithi077 2024-03-16T09:38:05.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:04 smithi077 ceph-mon[32171]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:38:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:04 smithi119 ceph-mon[35632]: Deploying daemon osd.9 on smithi077 2024-03-16T09:38:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:04 smithi119 ceph-mon[35632]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:38:05.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:04 smithi129 ceph-mon[35841]: Deploying daemon osd.9 on smithi077 2024-03-16T09:38:05.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:04 smithi129 ceph-mon[35841]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:38:07.093 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:06 smithi129 ceph-mon[35841]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-16T09:38:07.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:06 smithi119 ceph-mon[35632]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-16T09:38:07.193 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:06 smithi077 ceph-mon[32171]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-16T09:38:09.093 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:08 smithi077 ceph-mon[32171]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:38:09.093 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:09.094 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:09.094 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:08 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:08 smithi119 ceph-mon[35632]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:38:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:08 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:09.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:08 smithi129 ceph-mon[35841]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:38:09.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:09.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:09.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:08 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:10.898 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:10 smithi077 ceph-mon[32171]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:11.103 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:10 smithi129 ceph-mon[35841]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:11.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:10 smithi119 ceph-mon[35632]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:12.010 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 9 on host 'smithi077' 2024-03-16T09:38:12.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.007+0000 7f7b0cff9700 1 -- 172.21.15.77:0/3020338070 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7b10062710 con 0x7f7afc0a2890 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7afc0a2890 msgr2=0x7f7afc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7afc0a2890 0x7f7afc0a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7af8005950 tx=0x7f7af80058e0 comp rx=0 tx=0).stop 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 msgr2=0x7f7b10100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10100d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000b5c0 tx=0x7f7b000047b0 comp rx=0 tx=0).stop 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 shutdown_connections 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7afc0a2890 0x7f7afc0a4d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b10107ff0 0x7f7b10101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b10102d90 0x7f7b101026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 --2- 172.21.15.77:0/3020338070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b101056c0 0x7f7b10100d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 >> 172.21.15.77:0/3020338070 conn(0x7f7b100fc7c0 msgr2=0x7f7b101036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:38:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 shutdown_connections 2024-03-16T09:38:12.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:38:12.010+0000 7f7b16168700 1 -- 172.21.15.77:0/3020338070 wait complete. 2024-03-16T09:38:12.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.639 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:12.669 DEBUG:teuthology.orchestra.run.smithi077:osd.9> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.9.service 2024-03-16T09:38:12.672 INFO:tasks.cephadm:Deploying osd.10 on smithi119 with /dev/vg_nvme/lv_1... 2024-03-16T09:38:12.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:12.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:12.906 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:38:13.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:13 smithi129 ceph-mon[35841]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T09:38:13.689 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:13 smithi119 ceph-mon[35632]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T09:38:13.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:13 smithi077 ceph-mon[32171]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T09:38:14.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:14 smithi077 ceph-mon[32171]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T09:38:14.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:14 smithi077 ceph-mon[32171]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T09:38:14.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:14 smithi077 ceph-mon[32171]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:38:14.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:14.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:14 smithi077 ceph-mon[32171]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:14.315 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:38:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:38:14.299+0000 7f6047e04700 -1 osd.9 0 waiting for initial osdmap 2024-03-16T09:38:14.315 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:38:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:38:14.311+0000 7f6043bf7700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:38:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:14 smithi129 ceph-mon[35841]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T09:38:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:14 smithi129 ceph-mon[35841]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T09:38:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:14 smithi129 ceph-mon[35841]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:38:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:14 smithi129 ceph-mon[35841]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:14.689 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:14 smithi119 ceph-mon[35632]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T09:38:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:14 smithi119 ceph-mon[35632]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T09:38:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:14 smithi119 ceph-mon[35632]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T09:38:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:14 smithi119 ceph-mon[35632]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:14.888 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T09:38:14.888 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T09:38:14.888 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-16T09:38:14.888 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-16T09:38:14.888 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011173 s, 938 MB/s 2024-03-16T09:38:14.888 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-16T09:38:15.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:15 smithi119 ceph-mon[35632]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:38:15.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:15 smithi119 ceph-mon[35632]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T09:38:15.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:15 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:15.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:15 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:15.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:15 smithi077 ceph-mon[32171]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:38:15.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:15 smithi077 ceph-mon[32171]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T09:38:15.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:15 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:15.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:15 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:15.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-03-16T09:38:15.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:15 smithi129 ceph-mon[35841]: from='osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T09:38:15.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:15 smithi129 ceph-mon[35841]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T09:38:15.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:15 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:15.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:15 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:15.801 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:38:16.513 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:38:16.513 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:38:16.513 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] boot 2024-03-16T09:38:16.513 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi077 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi077 to 4272M 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:16.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.515 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:16 smithi119 ceph-mon[35632]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] boot 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi077 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi077 to 4272M 2024-03-16T09:38:16.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:16.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:16.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:16 smithi077 ceph-mon[32171]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: osd.9 [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] boot 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi077 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi077 to 4272M 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:16.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:16 smithi129 ceph-mon[35841]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-03-16T09:38:17.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.406+0000 7f023fad5700 1 -- 172.21.15.119:0/451451093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102400 msgr2=0x7f0238102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:17.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.406+0000 7f023fad5700 1 --2- 172.21.15.119:0/451451093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102400 0x7f0238102860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f02340099c0 tx=0x7f0234009cd0 comp rx=0 tx=0).stop 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.406+0000 7f023fad5700 1 -- 172.21.15.119:0/451451093 shutdown_connections 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.406+0000 7f023fad5700 1 --2- 172.21.15.119:0/451451093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0238102da0 0x7f0238109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.406+0000 7f023fad5700 1 --2- 172.21.15.119:0/451451093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0238101200 0x7f0238101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.406+0000 7f023fad5700 1 --2- 172.21.15.119:0/451451093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102400 0x7f0238102860 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.406+0000 7f023fad5700 1 -- 172.21.15.119:0/451451093 >> 172.21.15.119:0/451451093 conn(0x7f02380fc7c0 msgr2=0x7f02380febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 -- 172.21.15.119:0/451451093 shutdown_connections 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 -- 172.21.15.119:0/451451093 wait complete. 2024-03-16T09:38:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 Processor -- start 2024-03-16T09:38:17.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 -- start start 2024-03-16T09:38:17.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0238101200 0x7f023806b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:17.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0238102400 0x7f023819d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:17.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102da0 0x7f023819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:17.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f023810cc90 con 0x7f0238101200 2024-03-16T09:38:17.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f023810cb10 con 0x7f0238102da0 2024-03-16T09:38:17.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023fad5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f023810ce10 con 0x7f0238102400 2024-03-16T09:38:17.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023e072700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102da0 0x7f023819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:17.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023e072700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102da0 0x7f023819f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55190/0 (socket says 172.21.15.119:55190) 2024-03-16T09:38:17.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.407+0000 7f023e072700 1 -- 172.21.15.119:0/398514903 learned_addr learned my addr 172.21.15.119:0/398514903 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-16T09:38:17.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023d871700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0238101200 0x7f023806b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:17.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023d070700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0238102400 0x7f023819d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:17.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023e072700 1 -- 172.21.15.119:0/398514903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0238102400 msgr2=0x7f023819d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023e072700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0238102400 0x7f023819d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023e072700 1 -- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0238101200 msgr2=0x7f023806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023e072700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0238101200 0x7f023806b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023e072700 1 -- 172.21.15.119:0/398514903 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02381a00f0 con 0x7f0238102da0 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023e072700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102da0 0x7f023819f820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f023000ada0 tx=0x7f023000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f022effd700 1 -- 172.21.15.119:0/398514903 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230017020 con 0x7f0238102da0 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f022effd700 1 -- 172.21.15.119:0/398514903 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f023000ce30 con 0x7f0238102da0 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f022effd700 1 -- 172.21.15.119:0/398514903 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230005350 con 0x7f0238102da0 2024-03-16T09:38:17.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02381a03e0 con 0x7f0238102da0 2024-03-16T09:38:17.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.408+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02381a08d0 con 0x7f0238102da0 2024-03-16T09:38:17.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.409+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0238068f90 con 0x7f0238102da0 2024-03-16T09:38:17.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.410+0000 7f022effd700 1 -- 172.21.15.119:0/398514903 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f023000c980 con 0x7f0238102da0 2024-03-16T09:38:17.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.411+0000 7f022effd700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f02240a2890 0x7f02240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:17.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.411+0000 7f022effd700 1 -- 172.21.15.119:0/398514903 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f02300cdfe0 con 0x7f0238102da0 2024-03-16T09:38:17.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.411+0000 7f023d871700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f02240a2890 0x7f02240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:17.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.412+0000 7f023d871700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f02240a2890 0x7f02240a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0238102260 tx=0x7f022800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:38:17.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.415+0000 7f022effd700 1 -- 172.21.15.119:0/398514903 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0230021020 con 0x7f0238102da0 2024-03-16T09:38:17.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:17 smithi119 ceph-mon[35632]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T09:38:17.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:17.618+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0238062710 con 0x7f02240a2890 2024-03-16T09:38:17.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:17 smithi077 ceph-mon[32171]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T09:38:17.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:17 smithi129 ceph-mon[35841]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T09:38:18.533 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:18 smithi119 ceph-mon[35632]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:38:18.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:38:18.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:38:18.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:18.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:18 smithi119 ceph-mon[35632]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:18.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:18 smithi077 ceph-mon[32171]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:38:18.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:38:18.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:38:18.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:18.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:18 smithi077 ceph-mon[32171]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:18.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:18 smithi129 ceph-mon[35841]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:38:18.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:38:18.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:38:18.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:18.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:18 smithi129 ceph-mon[35841]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:20.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:20 smithi119 ceph-mon[35632]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:20.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:20 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/1427253552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]: dispatch 2024-03-16T09:38:20.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:20 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]: dispatch 2024-03-16T09:38:20.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:20 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]': finished 2024-03-16T09:38:20.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:20 smithi119 ceph-mon[35632]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T09:38:20.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:20 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:20 smithi077 ceph-mon[32171]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:20 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/1427253552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]: dispatch 2024-03-16T09:38:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:20 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]: dispatch 2024-03-16T09:38:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:20 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]': finished 2024-03-16T09:38:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:20 smithi077 ceph-mon[32171]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T09:38:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:20 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:21.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:20 smithi129 ceph-mon[35841]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:20 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/1427253552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]: dispatch 2024-03-16T09:38:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:20 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]: dispatch 2024-03-16T09:38:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:20 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68f623f5-73ff-46d2-8ff8-30767522cc80"}]': finished 2024-03-16T09:38:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:20 smithi129 ceph-mon[35841]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T09:38:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:20 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:21 smithi119 ceph-mon[35632]: from='client.? 172.21.15.119:0/1814536940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:38:22.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:21 smithi077 ceph-mon[32171]: from='client.? 172.21.15.119:0/1814536940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:38:22.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:21 smithi129 ceph-mon[35841]: from='client.? 172.21.15.119:0/1814536940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:38:23.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:22 smithi119 ceph-mon[35632]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:23.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:22 smithi077 ceph-mon[32171]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:23.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:22 smithi129 ceph-mon[35841]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:25.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:24 smithi119 ceph-mon[35632]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:25.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:24 smithi077 ceph-mon[32171]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:25.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:24 smithi129 ceph-mon[35841]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:26.923 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:26 smithi119 ceph-mon[35632]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:27.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:26 smithi077 ceph-mon[32171]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:27.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:26 smithi129 ceph-mon[35841]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:28.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:28 smithi119 ceph-mon[35632]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:28.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T09:38:28.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:28 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:28 smithi077 ceph-mon[32171]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T09:38:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:28 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:29.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:28 smithi129 ceph-mon[35841]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T09:38:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:28 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:29.929 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:29 smithi119 ceph-mon[35632]: Deploying daemon osd.10 on smithi119 2024-03-16T09:38:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:29 smithi077 ceph-mon[32171]: Deploying daemon osd.10 on smithi119 2024-03-16T09:38:30.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:29 smithi129 ceph-mon[35841]: Deploying daemon osd.10 on smithi119 2024-03-16T09:38:30.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:30 smithi119 ceph-mon[35632]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:30 smithi077 ceph-mon[32171]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:30 smithi129 ceph-mon[35841]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:32.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:32 smithi119 ceph-mon[35632]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:32 smithi077 ceph-mon[32171]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:33.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:32 smithi129 ceph-mon[35841]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:34.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:34.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:34.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:34.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:34 smithi077 ceph-mon[32171]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:34.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:34.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:34.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:34.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:34 smithi129 ceph-mon[35841]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:34.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:34.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:34.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:34.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:34 smithi119 ceph-mon[35632]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:37.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:36 smithi119 ceph-mon[35632]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:37.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:36 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:36 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:36 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:37.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:36 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:37.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:36 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.090 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 10 on host 'smithi119' 2024-03-16T09:38:37.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.088+0000 7f022effd700 1 -- 172.21.15.119:0/398514903 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0238062710 con 0x7f02240a2890 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f02240a2890 msgr2=0x7f02240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f02240a2890 0x7f02240a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0238102260 tx=0x7f022800b410 comp rx=0 tx=0).stop 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102da0 msgr2=0x7f023819f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102da0 0x7f023819f820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f023000ada0 tx=0x7f023000c330 comp rx=0 tx=0).stop 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 shutdown_connections 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f02240a2890 0x7f02240a4d50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0238102da0 0x7f023819f820 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0238101200 0x7f023806b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 --2- 172.21.15.119:0/398514903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0238102400 0x7f023819d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 >> 172.21.15.119:0/398514903 conn(0x7f02380fc7c0 msgr2=0x7f0238105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 shutdown_connections 2024-03-16T09:38:37.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-16T09:38:37.090+0000 7f023fad5700 1 -- 172.21.15.119:0/398514903 wait complete. 2024-03-16T09:38:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:36 smithi077 ceph-mon[32171]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:36 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:36 smithi129 ceph-mon[35841]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:36 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:36 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:36 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:36 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:36 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:37.759 DEBUG:teuthology.orchestra.run.smithi119:osd.10> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.10.service 2024-03-16T09:38:37.761 INFO:tasks.cephadm:Deploying osd.11 on smithi129 with /dev/vg_nvme/lv_1... 2024-03-16T09:38:37.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T09:38:38.013 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:38:38.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:38 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:38.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:38 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:38.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:38 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:38.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:38 smithi129 ceph-mon[35841]: from='osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T09:38:38.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:38 smithi129 ceph-mon[35841]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T09:38:38.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:38 smithi129 ceph-mon[35841]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:38.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:38 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:38.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:38 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:38.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:38 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:38.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:38 smithi119 ceph-mon[35632]: from='osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T09:38:38.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:38 smithi119 ceph-mon[35632]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T09:38:38.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:38 smithi119 ceph-mon[35632]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:38.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:38.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:38.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:38 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:38.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:38 smithi077 ceph-mon[32171]: from='osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T09:38:38.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:38 smithi077 ceph-mon[32171]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T09:38:38.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:38 smithi077 ceph-mon[32171]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:39 smithi119 ceph-mon[35632]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T09:38:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:39 smithi119 ceph-mon[35632]: from='osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:38:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:39 smithi119 ceph-mon[35632]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T09:38:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:39 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:39 smithi119 ceph-mon[35632]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:38:39.190 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:38:39 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:38:39.056+0000 7ff6698de700 -1 osd.10 0 waiting for initial osdmap 2024-03-16T09:38:39.190 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:38:39 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:38:39.070+0000 7ff6656d1700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:38:39.380 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:39 smithi129 ceph-mon[35841]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T09:38:39.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:39 smithi129 ceph-mon[35841]: from='osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:38:39.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:39 smithi129 ceph-mon[35841]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T09:38:39.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:39 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:39.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:39 smithi129 ceph-mon[35841]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:38:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:39 smithi077 ceph-mon[32171]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T09:38:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:39 smithi077 ceph-mon[32171]: from='osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:38:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:39 smithi077 ceph-mon[32171]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T09:38:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:39 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:39 smithi077 ceph-mon[32171]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-16T09:38:39.977 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T09:38:39.977 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T09:38:39.977 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-16T09:38:39.978 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-16T09:38:39.978 INFO:teuthology.orchestra.run.smithi129.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113741 s, 922 MB/s 2024-03-16T09:38:39.978 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] boot 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T09:38:40.305 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] boot 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T09:38:40.449 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-16T09:38:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T09:38:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-03-16T09:38:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: osd.10 [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] boot 2024-03-16T09:38:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T09:38:40.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T09:38:40.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2024-03-16T09:38:40.914 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi119 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-16T09:38:41.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:41.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:41.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:41 smithi129 ceph-mon[35841]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T09:38:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:38:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:38:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi119 2024-03-16T09:38:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:41 smithi119 ceph-mon[35632]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi119 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:38:41.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-16T09:38:41.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:41.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:38:41.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:41.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:41 smithi077 ceph-mon[32171]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T09:38:42.253 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:42 smithi129 ceph-mon[35841]: pgmap v236: 1 pgs: 1 remapped+peering; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:42.253 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:42 smithi129 ceph-mon[35841]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T09:38:42.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:42 smithi119 ceph-mon[35632]: pgmap v236: 1 pgs: 1 remapped+peering; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:42.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:42 smithi119 ceph-mon[35632]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T09:38:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.493+0000 7f9d20729700 1 -- 172.21.15.129:0/1744931032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 msgr2=0x7f9d18103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.493+0000 7f9d20729700 1 --2- 172.21.15.129:0/1744931032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18103960 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009990 tx=0x7f9d08009ca0 comp rx=0 tx=0).stop 2024-03-16T09:38:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 -- 172.21.15.129:0/1744931032 shutdown_connections 2024-03-16T09:38:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 --2- 172.21.15.129:0/1744931032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d18102300 0x7f9d18102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 --2- 172.21.15.129:0/1744931032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d18103ea0 0x7f9d1810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 --2- 172.21.15.129:0/1744931032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18103960 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 -- 172.21.15.129:0/1744931032 >> 172.21.15.129:0/1744931032 conn(0x7f9d180fd8e0 msgr2=0x7f9d180ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 -- 172.21.15.129:0/1744931032 shutdown_connections 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 -- 172.21.15.129:0/1744931032 wait complete. 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 Processor -- start 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 -- start start 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d18102300 0x7f9d18198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.494+0000 7f9d20729700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d20729700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d18103ea0 0x7f9d1819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d20729700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1810de10 con 0x7f9d18103ea0 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d20729700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1810dc90 con 0x7f9d18102300 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d20729700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1810df90 con 0x7f9d18103500 2024-03-16T09:38:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57030/0 (socket says 172.21.15.129:57030) 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 -- 172.21.15.129:0/2429364908 learned_addr learned my addr 172.21.15.129:0/2429364908 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1ecc6700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d18103ea0 0x7f9d1819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1e4c5700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d18102300 0x7f9d18198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 -- 172.21.15.129:0/2429364908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d18102300 msgr2=0x7f9d18198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d18102300 0x7f9d18198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 -- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d18103ea0 msgr2=0x7f9d1819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d18103ea0 0x7f9d1819fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 -- 172.21.15.129:0/2429364908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d181a0150 con 0x7f9d18103500 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1e4c5700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d18102300 0x7f9d18198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d1dcc4700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18198a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9d08004790 tx=0x7f9d08009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d0f7fe700 1 -- 172.21.15.129:0/2429364908 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d08027070 con 0x7f9d18103500 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d181a03e0 con 0x7f9d18103500 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d0f7fe700 1 -- 172.21.15.129:0/2429364908 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d08004140 con 0x7f9d18103500 2024-03-16T09:38:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d0f7fe700 1 -- 172.21.15.129:0/2429364908 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d08022d70 con 0x7f9d18103500 2024-03-16T09:38:42.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.495+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d181a08b0 con 0x7f9d18103500 2024-03-16T09:38:42.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.496+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d18068f90 con 0x7f9d18103500 2024-03-16T09:38:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.497+0000 7f9d0f7fe700 1 -- 172.21.15.129:0/2429364908 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9d0802a070 con 0x7f9d18103500 2024-03-16T09:38:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.498+0000 7f9d0f7fe700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9d040a2890 0x7f9d040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:38:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.498+0000 7f9d0f7fe700 1 -- 172.21.15.129:0/2429364908 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7f9d080cfa80 con 0x7f9d18103500 2024-03-16T09:38:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.498+0000 7f9d1e4c5700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9d040a2890 0x7f9d040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:38:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.499+0000 7f9d1e4c5700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9d040a2890 0x7f9d040a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d18103360 tx=0x7f9d14008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:38:42.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.501+0000 7f9d0f7fe700 1 -- 172.21.15.129:0/2429364908 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d08031070 con 0x7f9d18103500 2024-03-16T09:38:42.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:42 smithi077 ceph-mon[32171]: pgmap v236: 1 pgs: 1 remapped+peering; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:42.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:42 smithi077 ceph-mon[32171]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T09:38:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:38:42.708+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9d18062710 con 0x7f9d040a2890 2024-03-16T09:38:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:43 smithi119 ceph-mon[35632]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T09:38:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:43 smithi119 ceph-mon[35632]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:38:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:43 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:38:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:43 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:38:43.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:43 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:43 smithi077 ceph-mon[32171]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T09:38:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:43 smithi077 ceph-mon[32171]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:38:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:43 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:38:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:43 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:38:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:43 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:43 smithi129 ceph-mon[35841]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T09:38:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:43 smithi129 ceph-mon[35841]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:38:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T09:38:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T09:38:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:44.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:44 smithi129 ceph-mon[35841]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T09:38:44.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:44 smithi129 ceph-mon[35841]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:45.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:44 smithi077 ceph-mon[32171]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T09:38:45.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:44 smithi077 ceph-mon[32171]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:44 smithi119 ceph-mon[35632]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T09:38:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:44 smithi119 ceph-mon[35632]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:45 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/1545585559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]: dispatch 2024-03-16T09:38:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:45 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]: dispatch 2024-03-16T09:38:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:45 smithi077 ceph-mon[32171]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]': finished 2024-03-16T09:38:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:45 smithi077 ceph-mon[32171]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T09:38:46.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:45 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:38:46.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:45 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/1545585559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]: dispatch 2024-03-16T09:38:46.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:45 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]: dispatch 2024-03-16T09:38:46.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:45 smithi129 ceph-mon[35841]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]': finished 2024-03-16T09:38:46.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:45 smithi129 ceph-mon[35841]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T09:38:46.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:45 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:38:46.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:45 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/1545585559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]: dispatch 2024-03-16T09:38:46.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:45 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]: dispatch 2024-03-16T09:38:46.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:45 smithi119 ceph-mon[35632]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef748c28-d638-4d1d-a084-080ef7953c98"}]': finished 2024-03-16T09:38:46.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:45 smithi119 ceph-mon[35632]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T09:38:46.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:45 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:38:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:46 smithi077 ceph-mon[32171]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:47.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:46 smithi077 ceph-mon[32171]: from='client.? 172.21.15.129:0/2737867555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:38:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:46 smithi129 ceph-mon[35841]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:46 smithi129 ceph-mon[35841]: from='client.? 172.21.15.129:0/2737867555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:38:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:46 smithi119 ceph-mon[35632]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:38:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:46 smithi119 ceph-mon[35632]: from='client.? 172.21.15.129:0/2737867555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T09:38:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:48 smithi119 ceph-mon[35632]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-16T09:38:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:48 smithi077 ceph-mon[32171]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-16T09:38:49.288 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:48 smithi129 ceph-mon[35841]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-16T09:38:50.876 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:50 smithi129 ceph-mon[35841]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:38:51.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:50 smithi119 ceph-mon[35632]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:38:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:50 smithi077 ceph-mon[32171]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T09:38:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:52 smithi119 ceph-mon[35632]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-16T09:38:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:52 smithi077 ceph-mon[32171]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-16T09:38:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:52 smithi129 ceph-mon[35841]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-16T09:38:54.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:53 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T09:38:54.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:53 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:54.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:53 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T09:38:54.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:53 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:53 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T09:38:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:53 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:38:54.921 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:54 smithi129 ceph-mon[35841]: Deploying daemon osd.11 on smithi129 2024-03-16T09:38:54.921 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:54 smithi129 ceph-mon[35841]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:38:55.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:54 smithi077 ceph-mon[32171]: Deploying daemon osd.11 on smithi129 2024-03-16T09:38:55.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:54 smithi077 ceph-mon[32171]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:38:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:54 smithi119 ceph-mon[35632]: Deploying daemon osd.11 on smithi129 2024-03-16T09:38:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:54 smithi119 ceph-mon[35632]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T09:38:57.048 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:56 smithi129 ceph-mon[35841]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-16T09:38:57.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:56 smithi119 ceph-mon[35632]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-16T09:38:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:56 smithi077 ceph-mon[32171]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-16T09:38:59.167 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:58 smithi129 ceph-mon[35841]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:38:59.167 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:59.167 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:59.167 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:38:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:58 smithi119 ceph-mon[35632]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:38:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:38:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:38:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:58 smithi077 ceph-mon[32171]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T09:38:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:38:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:38:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:00.962 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:00 smithi129 ceph-mon[35841]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:00 smithi119 ceph-mon[35632]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:00 smithi077 ceph-mon[32171]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:02.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.627+0000 7f9d0f7fe700 1 -- 172.21.15.129:0/2429364908 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9d18062710 con 0x7f9d040a2890 2024-03-16T09:39:02.631 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 11 on host 'smithi129' 2024-03-16T09:39:02.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9d040a2890 msgr2=0x7f9d040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:02.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9d040a2890 0x7f9d040a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d18103360 tx=0x7f9d14008040 comp rx=0 tx=0).stop 2024-03-16T09:39:02.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 msgr2=0x7f9d18198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:02.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18198a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9d08004790 tx=0x7f9d08009580 comp rx=0 tx=0).stop 2024-03-16T09:39:02.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 shutdown_connections 2024-03-16T09:39:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9d040a2890 0x7f9d040a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d18102300 0x7f9d18198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d18103ea0 0x7f9d1819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 --2- 172.21.15.129:0/2429364908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d18103500 0x7f9d18198a80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.630+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 >> 172.21.15.129:0/2429364908 conn(0x7f9d180fd8e0 msgr2=0x7f9d18106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.631+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 shutdown_connections 2024-03-16T09:39:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:02.631+0000 7f9d20729700 1 -- 172.21.15.129:0/2429364908 wait complete. 2024-03-16T09:39:02.995 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:02.995 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:02.995 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:02.995 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:02.995 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:02.995 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:02.995 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:02.996 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:02.996 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:02.996 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T09:39:02.996 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:02 smithi129 ceph-mon[35841]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T09:39:03.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:02 smithi119 ceph-mon[35632]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T09:39:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:03.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:03.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T09:39:03.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:02 smithi077 ceph-mon[32171]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T09:39:03.379 DEBUG:teuthology.orchestra.run.smithi129:osd.11> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.11.service 2024-03-16T09:39:03.381 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-16T09:39:03.381 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T09:39:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:39:04.861 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:04 smithi129 ceph-mon[35841]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T09:39:04.861 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:04 smithi129 ceph-mon[35841]: from='osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:39:04.861 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:04 smithi129 ceph-mon[35841]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T09:39:04.861 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:04 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:04.861 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:04 smithi129 ceph-mon[35841]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:39:04.861 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:04 smithi129 ceph-mon[35841]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:04.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:04 smithi119 ceph-mon[35632]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T09:39:04.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:04 smithi119 ceph-mon[35632]: from='osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:39:04.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:04 smithi119 ceph-mon[35632]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T09:39:04.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:04 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:04.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:04 smithi119 ceph-mon[35632]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:39:04.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:04 smithi119 ceph-mon[35632]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:04.984 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:04 smithi077 ceph-mon[32171]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T09:39:04.985 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:04 smithi077 ceph-mon[32171]: from='osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:39:04.985 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:04 smithi077 ceph-mon[32171]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T09:39:04.985 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:04 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:04.985 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:04 smithi077 ceph-mon[32171]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-16T09:39:04.985 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:04 smithi077 ceph-mon[32171]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.312+0000 7fe433a64700 1 -- 172.21.15.77:0/3089136210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c101490 msgr2=0x7fe42c1018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.312+0000 7fe433a64700 1 --2- 172.21.15.77:0/3089136210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c101490 0x7fe42c1018f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009990 tx=0x7fe41c009ca0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- 172.21.15.77:0/3089136210 shutdown_connections 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 --2- 172.21.15.77:0/3089136210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42c1027e0 0x7fe42c102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 --2- 172.21.15.77:0/3089136210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c101e30 0x7fe42c1096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 --2- 172.21.15.77:0/3089136210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c101490 0x7fe42c1018f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- 172.21.15.77:0/3089136210 >> 172.21.15.77:0/3089136210 conn(0x7fe42c0fbf40 msgr2=0x7fe42c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- 172.21.15.77:0/3089136210 shutdown_connections 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- 172.21.15.77:0/3089136210 wait complete. 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 Processor -- start 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- start start 2024-03-16T09:39:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c101490 0x7fe42c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42c101e30 0x7fe42c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c1027e0 0x7fe42c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c10cca0 con 0x7fe42c1027e0 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c10cb20 con 0x7fe42c101e30 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.313+0000 7fe433a64700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c10ce20 con 0x7fe42c101490 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c1027e0 0x7fe42c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe431800700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c101490 0x7fe42c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c1027e0 0x7fe42c19f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43670/0 (socket says 172.21.15.77:43670) 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 -- 172.21.15.77:0/3999104120 learned_addr learned my addr 172.21.15.77:0/3999104120 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe430fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42c101e30 0x7fe42c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 -- 172.21.15.77:0/3999104120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c101490 msgr2=0x7fe42c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c101490 0x7fe42c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 -- 172.21.15.77:0/3999104120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42c101e30 msgr2=0x7fe42c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42c101e30 0x7fe42c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 -- 172.21.15.77:0/3999104120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c1a0070 con 0x7fe42c1027e0 2024-03-16T09:39:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe432001700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c1027e0 0x7fe42c19f9d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe42400ef60 tx=0x7fe42400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe4227fc700 1 -- 172.21.15.77:0/3999104120 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4240092e0 con 0x7fe42c1027e0 2024-03-16T09:39:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe4227fc700 1 -- 172.21.15.77:0/3999104120 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe424004bb0 con 0x7fe42c1027e0 2024-03-16T09:39:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42c1a0300 con 0x7fe42c1027e0 2024-03-16T09:39:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe4227fc700 1 -- 172.21.15.77:0/3999104120 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424004020 con 0x7fe42c1027e0 2024-03-16T09:39:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.314+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42c1a0890 con 0x7fe42c1027e0 2024-03-16T09:39:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.316+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42c068f90 con 0x7fe42c1027e0 2024-03-16T09:39:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.316+0000 7fe4227fc700 1 -- 172.21.15.77:0/3999104120 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe42401b030 con 0x7fe42c1027e0 2024-03-16T09:39:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.317+0000 7fe4227fc700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4180a2910 0x7fe4180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.317+0000 7fe431800700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4180a2910 0x7fe4180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.317+0000 7fe4227fc700 1 -- 172.21.15.77:0/3999104120 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7fe4240cde10 con 0x7fe42c1027e0 2024-03-16T09:39:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.318+0000 7fe431800700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4180a2910 0x7fe4180a4dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe42c069d40 tx=0x7fe428009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:05.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.322+0000 7fe4227fc700 1 -- 172.21.15.77:0/3999104120 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe424022020 con 0x7fe42c1027e0 2024-03-16T09:39:05.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.522+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fe42c1a0cd0 con 0x7fe42c1027e0 2024-03-16T09:39:05.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.523+0000 7fe4227fc700 1 -- 172.21.15.77:0/3999104120 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fe424010040 con 0x7fe42c1027e0 2024-03-16T09:39:05.526 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:05.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4180a2910 msgr2=0x7fe4180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:05.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4180a2910 0x7fe4180a4dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe42c069d40 tx=0x7fe428009260 comp rx=0 tx=0).stop 2024-03-16T09:39:05.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c1027e0 msgr2=0x7fe42c19f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:05.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c1027e0 0x7fe42c19f9d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe42400ef60 tx=0x7fe42400ef90 comp rx=0 tx=0).stop 2024-03-16T09:39:05.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 shutdown_connections 2024-03-16T09:39:05.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4180a2910 0x7fe4180a4dd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe42c1027e0 0x7fe42c19f9d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe42c101490 0x7fe42c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 --2- 172.21.15.77:0/3999104120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42c101e30 0x7fe42c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:05.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 >> 172.21.15.77:0/3999104120 conn(0x7fe42c0fbf40 msgr2=0x7fe42c100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:05.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 shutdown_connections 2024-03-16T09:39:05.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:05.525+0000 7fe433a64700 1 -- 172.21.15.77:0/3999104120 wait complete. 2024-03-16T09:39:05.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-mon[35841]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:39:05.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-mon[35841]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T09:39:05.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:05.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:05.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-mon[35841]: from='osd.11 ' entity='osd.11' 2024-03-16T09:39:05.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3999104120' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T09:39:05.801 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:39:05.514+0000 7fa28adad700 -1 osd.11 0 waiting for initial osdmap 2024-03-16T09:39:05.801 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:39:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:39:05.623+0000 7fa282b98700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T09:39:05.824 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:05 smithi077 ceph-mon[32171]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:39:05.824 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:05 smithi077 ceph-mon[32171]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T09:39:05.824 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:05 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:05.824 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:05 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:05.824 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:05 smithi077 ceph-mon[32171]: from='osd.11 ' entity='osd.11' 2024-03-16T09:39:05.824 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:05 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3999104120' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T09:39:05.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:05 smithi119 ceph-mon[35632]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-16T09:39:05.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:05 smithi119 ceph-mon[35632]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T09:39:05.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:05 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:05.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:05 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:05.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:05 smithi119 ceph-mon[35632]: from='osd.11 ' entity='osd.11' 2024-03-16T09:39:05.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:05 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3999104120' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T09:39:06.158 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1710581920,"num_in_osds":12,"osd_in_since":1710581925,"num_remapped_pgs":0} 2024-03-16T09:39:06.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:06 smithi077 ceph-mon[32171]: purged_snaps scrub starts 2024-03-16T09:39:06.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:06 smithi077 ceph-mon[32171]: purged_snaps scrub ok 2024-03-16T09:39:06.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:06 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:06.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:06 smithi077 ceph-mon[32171]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:06 smithi119 ceph-mon[35632]: purged_snaps scrub starts 2024-03-16T09:39:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:06 smithi119 ceph-mon[35632]: purged_snaps scrub ok 2024-03-16T09:39:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:06 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:06.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:06 smithi119 ceph-mon[35632]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:07.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:06 smithi129 ceph-mon[35841]: purged_snaps scrub starts 2024-03-16T09:39:07.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:06 smithi129 ceph-mon[35841]: purged_snaps scrub ok 2024-03-16T09:39:07.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:06 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:07.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:06 smithi129 ceph-mon[35841]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-03-16T09:39:07.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T09:39:07.399 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] boot 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: Detected new or changed devices on smithi129 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: Adjusting osd_memory_target on smithi129 to 4272M 2024-03-16T09:39:07.854 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:07.854 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:07.854 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:07.854 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:07 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] boot 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: Detected new or changed devices on smithi129 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:07.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: Adjusting osd_memory_target on smithi129 to 4272M 2024-03-16T09:39:07.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:07.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:07.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:07.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:07 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: osd.11 [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] boot 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: Detected new or changed devices on smithi129 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: Adjusting osd_memory_target on smithi129 to 4272M 2024-03-16T09:39:08.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:08.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:08.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:08.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:07 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:08.846 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:08 smithi077 ceph-mon[32171]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T09:39:08.846 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:08 smithi077 ceph-mon[32171]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:08 smithi129 ceph-mon[35841]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T09:39:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:08 smithi129 ceph-mon[35841]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 -- 172.21.15.77:0/817774322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8102330 msgr2=0x7f1af8102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 --2- 172.21.15.77:0/817774322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8102330 0x7f1af8102750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1af40099c0 tx=0x7f1af4009cd0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 -- 172.21.15.77:0/817774322 shutdown_connections 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 --2- 172.21.15.77:0/817774322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1af8103ed0 0x7f1af810a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 --2- 172.21.15.77:0/817774322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8102330 0x7f1af8102750 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 --2- 172.21.15.77:0/817774322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1af8103530 0x7f1af8103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 -- 172.21.15.77:0/817774322 >> 172.21.15.77:0/817774322 conn(0x7f1af80fd8d0 msgr2=0x7f1af80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 -- 172.21.15.77:0/817774322 shutdown_connections 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.093+0000 7f1afff79700 1 -- 172.21.15.77:0/817774322 wait complete. 2024-03-16T09:39:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 Processor -- start 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 -- start start 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1af8102330 0x7f1af8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8103530 0x7f1af81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1af8103ed0 0x7f1af819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810de20 con 0x7f1af8103530 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810dca0 con 0x7f1af8103ed0 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afff79700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810dfa0 con 0x7f1af8102330 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8103530 0x7f1af81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8103530 0x7f1af81989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39802/0 (socket says 172.21.15.77:39802) 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 -- 172.21.15.77:0/3272771354 learned_addr learned my addr 172.21.15.77:0/3272771354 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afe516700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1af8103ed0 0x7f1af819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afdd15700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1af8102330 0x7f1af8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 -- 172.21.15.77:0/3272771354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1af8102330 msgr2=0x7f1af8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1af8102330 0x7f1af8198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 -- 172.21.15.77:0/3272771354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1af8103ed0 msgr2=0x7f1af819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1af8103ed0 0x7f1af819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afd514700 1 -- 172.21.15.77:0/3272771354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af81a00c0 con 0x7f1af8103530 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.094+0000 7f1afe516700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1af8103ed0 0x7f1af819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.095+0000 7f1afd514700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8103530 0x7f1af81989a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800b560 tx=0x7f1ae800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.095+0000 7f1aeeffd700 1 -- 172.21.15.77:0/3272771354 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8015020 con 0x7f1af8103530 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.095+0000 7f1aeeffd700 1 -- 172.21.15.77:0/3272771354 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ae8011ce0 con 0x7f1af8103530 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.095+0000 7f1aeeffd700 1 -- 172.21.15.77:0/3272771354 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8010da0 con 0x7f1af8103530 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.095+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af81a03b0 con 0x7f1af8103530 2024-03-16T09:39:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.095+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af81a0920 con 0x7f1af8103530 2024-03-16T09:39:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.096+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af8068f90 con 0x7f1af8103530 2024-03-16T09:39:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.098+0000 7f1aeeffd700 1 -- 172.21.15.77:0/3272771354 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1ae8018020 con 0x7f1af8103530 2024-03-16T09:39:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.098+0000 7f1aeeffd700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ae40a2890 0x7f1ae40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.098+0000 7f1afdd15700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ae40a2890 0x7f1ae40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.098+0000 7f1aeeffd700 1 -- 172.21.15.77:0/3272771354 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1ae80ce3d0 con 0x7f1af8103530 2024-03-16T09:39:09.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.099+0000 7f1afdd15700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ae40a2890 0x7f1ae40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1af40038f0 tx=0x7f1af4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:09.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.101+0000 7f1aeeffd700 1 -- 172.21.15.77:0/3272771354 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ae801f020 con 0x7f1af8103530 2024-03-16T09:39:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:08 smithi119 ceph-mon[35632]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T09:39:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:08 smithi119 ceph-mon[35632]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:09.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.293+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f1af802db70 con 0x7f1af8103530 2024-03-16T09:39:09.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.294+0000 7f1aeeffd700 1 -- 172.21.15.77:0/3272771354 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f1ae809b910 con 0x7f1af8103530 2024-03-16T09:39:09.297 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ae40a2890 msgr2=0x7f1ae40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ae40a2890 0x7f1ae40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1af40038f0 tx=0x7f1af4000bc0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8103530 msgr2=0x7f1af81989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8103530 0x7f1af81989a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800b560 tx=0x7f1ae800b920 comp rx=0 tx=0).stop 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 shutdown_connections 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ae40a2890 0x7f1ae40a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1af8103ed0 0x7f1af819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1af8102330 0x7f1af8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 --2- 172.21.15.77:0/3272771354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1af8103530 0x7f1af81989a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 >> 172.21.15.77:0/3272771354 conn(0x7f1af80fd8d0 msgr2=0x7f1af8106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 shutdown_connections 2024-03-16T09:39:09.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:09.296+0000 7f1afff79700 1 -- 172.21.15.77:0/3272771354 wait complete. 2024-03-16T09:39:09.941 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:09 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3272771354' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T09:39:09.971 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1710581946,"num_in_osds":12,"osd_in_since":1710581925,"num_remapped_pgs":0} 2024-03-16T09:39:09.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T09:39:10.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:09 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3272771354' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T09:39:10.091 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:09 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3272771354' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T09:39:10.212 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:39:11.005 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:10 smithi077 ceph-mon[32171]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:11.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:10 smithi129 ceph-mon[35841]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:11.107 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:10 smithi119 ceph-mon[35632]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:11.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 -- 172.21.15.77:0/3475737855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c102f60 msgr2=0x7fc16c106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:11.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 --2- 172.21.15.77:0/3475737855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c102f60 0x7fc16c106f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc1600099c0 tx=0x7fc160009cd0 comp rx=0 tx=0).stop 2024-03-16T09:39:11.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 -- 172.21.15.77:0/3475737855 shutdown_connections 2024-03-16T09:39:11.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 --2- 172.21.15.77:0/3475737855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc16c107660 0x7fc16c10f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:11.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 --2- 172.21.15.77:0/3475737855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc16c1025b0 0x7fc16c102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:11.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 --2- 172.21.15.77:0/3475737855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c102f60 0x7fc16c106f30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:11.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 -- 172.21.15.77:0/3475737855 >> 172.21.15.77:0/3475737855 conn(0x7fc16c0793f0 msgr2=0x7fc16c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:11.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 -- 172.21.15.77:0/3475737855 shutdown_connections 2024-03-16T09:39:11.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.914+0000 7fc17252c700 1 -- 172.21.15.77:0/3475737855 wait complete. 2024-03-16T09:39:11.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 Processor -- start 2024-03-16T09:39:11.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 -- start start 2024-03-16T09:39:11.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc16c1025b0 0x7fc16c0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:11.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc16c102f60 0x7fc16c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:11.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c107660 0x7fc16c075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:11.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c1124e0 con 0x7fc16c107660 2024-03-16T09:39:11.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c112360 con 0x7fc16c1025b0 2024-03-16T09:39:11.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc17252c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c112660 con 0x7fc16c102f60 2024-03-16T09:39:11.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc170ac9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c107660 0x7fc16c075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:11.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc170ac9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c107660 0x7fc16c075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39820/0 (socket says 172.21.15.77:39820) 2024-03-16T09:39:11.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc170ac9700 1 -- 172.21.15.77:0/3657301082 learned_addr learned my addr 172.21.15.77:0/3657301082 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:11.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc16bfff700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc16c1025b0 0x7fc16c0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:11.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc16b7fe700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc16c102f60 0x7fc16c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:11.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc170ac9700 1 -- 172.21.15.77:0/3657301082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc16c102f60 msgr2=0x7fc16c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:11.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc170ac9700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc16c102f60 0x7fc16c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:11.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.915+0000 7fc170ac9700 1 -- 172.21.15.77:0/3657301082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc16c1025b0 msgr2=0x7fc16c0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:11.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc170ac9700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc16c1025b0 0x7fc16c0745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:11.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc170ac9700 1 -- 172.21.15.77:0/3657301082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c1a6f80 con 0x7fc16c107660 2024-03-16T09:39:11.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc16bfff700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc16c1025b0 0x7fc16c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:39:11.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc170ac9700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c107660 0x7fc16c075050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc16400ef30 tx=0x7fc16400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:11.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc1697fa700 1 -- 172.21.15.77:0/3657301082 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1640092e0 con 0x7fc16c107660 2024-03-16T09:39:11.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc1697fa700 1 -- 172.21.15.77:0/3657301082 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc164004bb0 con 0x7fc16c107660 2024-03-16T09:39:11.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc1697fa700 1 -- 172.21.15.77:0/3657301082 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164004060 con 0x7fc16c107660 2024-03-16T09:39:11.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16c1a7210 con 0x7fc16c107660 2024-03-16T09:39:11.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.916+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16c1a7720 con 0x7fc16c107660 2024-03-16T09:39:11.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.917+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc16c10a830 con 0x7fc16c107660 2024-03-16T09:39:11.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.918+0000 7fc1697fa700 1 -- 172.21.15.77:0/3657301082 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc16401b030 con 0x7fc16c107660 2024-03-16T09:39:11.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.919+0000 7fc1697fa700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc1540a2910 0x7fc1540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:11.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.919+0000 7fc16bfff700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc1540a2910 0x7fc1540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:11.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.919+0000 7fc1697fa700 1 -- 172.21.15.77:0/3657301082 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc1640cea20 con 0x7fc16c107660 2024-03-16T09:39:11.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.919+0000 7fc16bfff700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc1540a2910 0x7fc1540a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc15c009de0 tx=0x7fc15c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:11.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:11.922+0000 7fc1697fa700 1 -- 172.21.15.77:0/3657301082 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc164022020 con 0x7fc16c107660 2024-03-16T09:39:12.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.119+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc16c02db70 con 0x7fc16c107660 2024-03-16T09:39:12.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.120+0000 7fc1697fa700 1 -- 172.21.15.77:0/3657301082 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18468 (secure 0 0 0) 0x7fc164010040 con 0x7fc16c107660 2024-03-16T09:39:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:12.124 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":82,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:07.686492+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:39:12.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc1540a2910 msgr2=0x7fc1540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:12.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc1540a2910 0x7fc1540a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc15c009de0 tx=0x7fc15c009190 comp rx=0 tx=0).stop 2024-03-16T09:39:12.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c107660 msgr2=0x7fc16c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:12.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c107660 0x7fc16c075050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc16400ef30 tx=0x7fc16400c3a0 comp rx=0 tx=0).stop 2024-03-16T09:39:12.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 shutdown_connections 2024-03-16T09:39:12.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc1540a2910 0x7fc1540a4dd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:12.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc16c1025b0 0x7fc16c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:12.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc16c107660 0x7fc16c075050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:12.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 --2- 172.21.15.77:0/3657301082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc16c102f60 0x7fc16c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:12.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 >> 172.21.15.77:0/3657301082 conn(0x7fc16c0793f0 msgr2=0x7fc16c108790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:12.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 shutdown_connections 2024-03-16T09:39:12.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:12.122+0000 7fc17252c700 1 -- 172.21.15.77:0/3657301082 wait complete. 2024-03-16T09:39:12.793 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:39:12.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-16T09:39:13.033 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:39:13.125 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:12 smithi129 ceph-mon[35841]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:13.125 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:12 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3657301082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:13.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:12 smithi077 ceph-mon[32171]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:13.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:12 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3657301082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:13.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:12 smithi119 ceph-mon[35632]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:12 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3657301082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 -- 172.21.15.77:0/3871094976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 msgr2=0x7fda001027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 --2- 172.21.15.77:0/3871094976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda001027a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc0099c0 tx=0x7fd9fc009cd0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 -- 172.21.15.77:0/3871094976 shutdown_connections 2024-03-16T09:39:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 --2- 172.21.15.77:0/3871094976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda00102ce0 0x7fda0010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 --2- 172.21.15.77:0/3871094976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda00108300 0x7fda001086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 --2- 172.21.15.77:0/3871094976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda001027a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 -- 172.21.15.77:0/3871094976 >> 172.21.15.77:0/3871094976 conn(0x7fda00079400 msgr2=0x7fda00079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.634+0000 7fda08403700 1 -- 172.21.15.77:0/3871094976 shutdown_connections 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 -- 172.21.15.77:0/3871094976 wait complete. 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 Processor -- start 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 -- start start 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda0019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda00102ce0 0x7fda0019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda00108300 0x7fda001987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fda00112450 con 0x7fda00102320 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fda001122d0 con 0x7fda00108300 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda08403700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fda001125d0 con 0x7fda00102ce0 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda0619f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda0019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda0619f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda0019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39850/0 (socket says 172.21.15.77:39850) 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda0619f700 1 -- 172.21.15.77:0/2789188627 learned_addr learned my addr 172.21.15.77:0/2789188627 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.635+0000 7fda0599e700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda00102ce0 0x7fda0019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:14.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda069a0700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda00108300 0x7fda001987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda0619f700 1 -- 172.21.15.77:0/2789188627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda00102ce0 msgr2=0x7fda0019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda0619f700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda00102ce0 0x7fda0019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda0619f700 1 -- 172.21.15.77:0/2789188627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda00108300 msgr2=0x7fda001987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda0619f700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda00108300 0x7fda001987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda0619f700 1 -- 172.21.15.77:0/2789188627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda00198f90 con 0x7fda00102320 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda0619f700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda0019e630 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000d980 tx=0x7fd9f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fd9f77fe700 1 -- 172.21.15.77:0/2789188627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0017070 con 0x7fda00102320 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fd9f77fe700 1 -- 172.21.15.77:0/2789188627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9f0005020 con 0x7fda00102320 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda00199280 con 0x7fda00102320 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fd9f77fe700 1 -- 172.21.15.77:0/2789188627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0009b50 con 0x7fda00102320 2024-03-16T09:39:14.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.636+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda001a5490 con 0x7fda00102320 2024-03-16T09:39:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.637+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda001039b0 con 0x7fda00102320 2024-03-16T09:39:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.638+0000 7fd9f77fe700 1 -- 172.21.15.77:0/2789188627 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd9f0009cf0 con 0x7fda00102320 2024-03-16T09:39:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.638+0000 7fd9f77fe700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9ec0a2890 0x7fd9ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.639+0000 7fda0599e700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9ec0a2890 0x7fd9ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.639+0000 7fd9f77fe700 1 -- 172.21.15.77:0/2789188627 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd9f00ce440 con 0x7fda00102320 2024-03-16T09:39:14.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.639+0000 7fda0599e700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9ec0a2890 0x7fd9ec0a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda00079b00 tx=0x7fd9fc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:14.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.643+0000 7fd9f77fe700 1 -- 172.21.15.77:0/2789188627 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9f0020020 con 0x7fda00102320 2024-03-16T09:39:14.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.837+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda0002db70 con 0x7fda00102320 2024-03-16T09:39:14.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.838+0000 7fd9f77fe700 1 -- 172.21.15.77:0/2789188627 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fd9f009b980 con 0x7fda00102320 2024-03-16T09:39:14.841 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9ec0a2890 msgr2=0x7fd9ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9ec0a2890 0x7fd9ec0a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda00079b00 tx=0x7fd9fc005fb0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 msgr2=0x7fda0019e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda0019e630 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000d980 tx=0x7fd9f000dc90 comp rx=0 tx=0).stop 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 shutdown_connections 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9ec0a2890 0x7fd9ec0a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda00108300 0x7fda001987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda00102320 0x7fda0019e630 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 --2- 172.21.15.77:0/2789188627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda00102ce0 0x7fda0019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:14.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 >> 172.21.15.77:0/2789188627 conn(0x7fda00079400 msgr2=0x7fda001010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:14.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 shutdown_connections 2024-03-16T09:39:14.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:14.840+0000 7fda08403700 1 -- 172.21.15.77:0/2789188627 wait complete. 2024-03-16T09:39:15.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:14 smithi077 ceph-mon[32171]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:15.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:14 smithi119 ceph-mon[35632]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:15.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:14 smithi129 ceph-mon[35841]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:15.464 INFO:tasks.cephadm:Adding mds.a on smithi077 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.d on smithi077 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.g on smithi077 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.j on smithi077 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.b on smithi119 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.e on smithi119 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.h on smithi119 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.k on smithi119 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.c on smithi129 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.f on smithi129 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.i on smithi129 2024-03-16T09:39:15.465 INFO:tasks.cephadm:Adding mds.l on smithi129 2024-03-16T09:39:15.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l' 2024-03-16T09:39:15.718 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:39:16.162 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:15 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2789188627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:16.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:15 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2789188627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:16.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:15 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2789188627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:17.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:16 smithi129 ceph-mon[35841]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:17.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:16 smithi119 ceph-mon[35632]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:16 smithi077 ceph-mon[32171]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:17.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 -- 172.21.15.129:0/3356133808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0102dd0 msgr2=0x7f96b01051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/3356133808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0102dd0 0x7f96b01051c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009990 tx=0x7f96a0009ca0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 -- 172.21.15.129:0/3356133808 shutdown_connections 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/3356133808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0108030 0x7f96b010a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/3356133808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0102dd0 0x7f96b01051c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/3356133808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96b0105700 0x7f96b0107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 -- 172.21.15.129:0/3356133808 >> 172.21.15.129:0/3356133808 conn(0x7f96b00fc7a0 msgr2=0x7f96b00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 -- 172.21.15.129:0/3356133808 shutdown_connections 2024-03-16T09:39:17.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.303+0000 7f96b4ebe700 1 -- 172.21.15.129:0/3356133808 wait complete. 2024-03-16T09:39:17.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 Processor -- start 2024-03-16T09:39:17.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 -- start start 2024-03-16T09:39:17.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0102dd0 0x7f96b01984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:17.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0105700 0x7f96b0198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:17.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96b0108030 0x7f96b019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010dd80 con 0x7f96b0102dd0 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010dc00 con 0x7f96b0108030 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010df00 con 0x7f96b0105700 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0105700 0x7f96b0198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0105700 0x7f96b0198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46494/0 (socket says 172.21.15.129:46494) 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 -- 172.21.15.129:0/550228091 learned_addr learned my addr 172.21.15.129:0/550228091 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96aed9d700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96b0108030 0x7f96b019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96ae59c700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0102dd0 0x7f96b01984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 -- 172.21.15.129:0/550228091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96b0108030 msgr2=0x7f96b019fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96b0108030 0x7f96b019fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 -- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0102dd0 msgr2=0x7f96b01984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0102dd0 0x7f96b01984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 -- 172.21.15.129:0/550228091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b01a0120 con 0x7f96b0105700 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96ae59c700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0102dd0 0x7f96b01984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96add9b700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0105700 0x7f96b0198a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f96a400d980 tx=0x7f96a400dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f969f7fe700 1 -- 172.21.15.129:0/550228091 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4016070 con 0x7f96b0105700 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.304+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b01a0410 con 0x7f96b0105700 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.305+0000 7f969f7fe700 1 -- 172.21.15.129:0/550228091 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96a4010460 con 0x7f96b0105700 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.305+0000 7f969f7fe700 1 -- 172.21.15.129:0/550228091 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a400f620 con 0x7f96b0105700 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.305+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b01a09a0 con 0x7f96b0105700 2024-03-16T09:39:17.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.305+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b0068f90 con 0x7f96b0105700 2024-03-16T09:39:17.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.307+0000 7f969f7fe700 1 -- 172.21.15.129:0/550228091 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f96a4019070 con 0x7f96b0105700 2024-03-16T09:39:17.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.307+0000 7f969f7fe700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f96980a2890 0x7f96980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:17.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.307+0000 7f969f7fe700 1 -- 172.21.15.129:0/550228091 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f96a40ce190 con 0x7f96b0105700 2024-03-16T09:39:17.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.307+0000 7f96ae59c700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f96980a2890 0x7f96980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:17.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.308+0000 7f96ae59c700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f96980a2890 0x7f96980a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96a00038b0 tx=0x7f96a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:17.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.311+0000 7f969f7fe700 1 -- 172.21.15.129:0/550228091 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96a409b6d0 con 0x7f96b0105700 2024-03-16T09:39:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.514+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l", "target": ["mon-mgr", ""]}) -- 0x7f96b00629d0 con 0x7f96980a2890 2024-03-16T09:39:17.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.522+0000 7f969f7fe700 1 -- 172.21.15.129:0/550228091 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f96b00629d0 con 0x7f96980a2890 2024-03-16T09:39:17.526 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mds.all update... 2024-03-16T09:39:17.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f96980a2890 msgr2=0x7f96980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:17.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f96980a2890 0x7f96980a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96a00038b0 tx=0x7f96a0005fb0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0105700 msgr2=0x7f96b0198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:17.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0105700 0x7f96b0198a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f96a400d980 tx=0x7f96a400dd40 comp rx=0 tx=0).stop 2024-03-16T09:39:17.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 shutdown_connections 2024-03-16T09:39:17.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f96980a2890 0x7f96980a4d50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96b0108030 0x7f96b019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0102dd0 0x7f96b01984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 --2- 172.21.15.129:0/550228091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96b0105700 0x7f96b0198a00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:17.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.524+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 >> 172.21.15.129:0/550228091 conn(0x7f96b00fc7a0 msgr2=0x7f96b01037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:17.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.525+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 shutdown_connections 2024-03-16T09:39:17.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-16T09:39:17.525+0000 7f96b4ebe700 1 -- 172.21.15.129:0/550228091 wait complete. 2024-03-16T09:39:18.144 DEBUG:teuthology.orchestra.run.smithi077:mds.a> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.a.service 2024-03-16T09:39:18.147 DEBUG:teuthology.orchestra.run.smithi077:mds.d> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.d.service 2024-03-16T09:39:18.190 DEBUG:teuthology.orchestra.run.smithi077:mds.g> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.g.service 2024-03-16T09:39:18.233 DEBUG:teuthology.orchestra.run.smithi077:mds.j> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.j.service 2024-03-16T09:39:18.277 DEBUG:teuthology.orchestra.run.smithi119:mds.b> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.b.service 2024-03-16T09:39:18.279 DEBUG:teuthology.orchestra.run.smithi119:mds.e> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.e.service 2024-03-16T09:39:18.323 DEBUG:teuthology.orchestra.run.smithi119:mds.h> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.h.service 2024-03-16T09:39:18.366 DEBUG:teuthology.orchestra.run.smithi119:mds.k> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.k.service 2024-03-16T09:39:18.409 DEBUG:teuthology.orchestra.run.smithi129:mds.c> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.c.service 2024-03-16T09:39:18.412 DEBUG:teuthology.orchestra.run.smithi129:mds.f> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.f.service 2024-03-16T09:39:18.455 DEBUG:teuthology.orchestra.run.smithi129:mds.i> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.i.service 2024-03-16T09:39:18.497 DEBUG:teuthology.orchestra.run.smithi129:mds.l> sudo journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.l.service 2024-03-16T09:39:18.540 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-16T09:39:18.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:39:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 -- 172.21.15.77:0/3218077605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 msgr2=0x7faaac1046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 --2- 172.21.15.77:0/3218077605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac1046f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009900 tx=0x7faa9c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 -- 172.21.15.77:0/3218077605 shutdown_connections 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 --2- 172.21.15.77:0/3218077605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faaac107560 0x7faaac1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 --2- 172.21.15.77:0/3218077605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaac104c30 0x7faaac107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 --2- 172.21.15.77:0/3218077605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac1046f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 -- 172.21.15.77:0/3218077605 >> 172.21.15.77:0/3218077605 conn(0x7faaac0fbd30 msgr2=0x7faaac0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 -- 172.21.15.77:0/3218077605 shutdown_connections 2024-03-16T09:39:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.649+0000 7faab117e640 1 -- 172.21.15.77:0/3218077605 wait complete. 2024-03-16T09:39:18.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.650+0000 7faab117e640 1 Processor -- start 2024-03-16T09:39:18.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.650+0000 7faab117e640 1 -- start start 2024-03-16T09:39:18.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faaac104c30 0x7faaac197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaac107560 0x7faaac19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faaac10d370 con 0x7faaac102300 2024-03-16T09:39:18.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7faaac10d1f0 con 0x7faaac104c30 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faaac10d4f0 con 0x7faaac107560 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56890/0 (socket says 172.21.15.77:56890) 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 -- 172.21.15.77:0/2712244637 learned_addr learned my addr 172.21.15.77:0/2712244637 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaa59c640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faaac104c30 0x7faaac197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaab59e640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaac107560 0x7faaac19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 -- 172.21.15.77:0/2712244637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaac107560 msgr2=0x7faaac19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaac107560 0x7faaac19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 -- 172.21.15.77:0/2712244637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faaac104c30 msgr2=0x7faaac197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faaac104c30 0x7faaac197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 -- 172.21.15.77:0/2712244637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac19f6c0 con 0x7faaac102300 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faaaad9d640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac197aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faa9c006010 tx=0x7faa9c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faa93fff640 1 -- 172.21.15.77:0/2712244637 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c043020 con 0x7faaac102300 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faa93fff640 1 -- 172.21.15.77:0/2712244637 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa9c0043d0 con 0x7faaac102300 2024-03-16T09:39:18.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaac19f950 con 0x7faaac102300 2024-03-16T09:39:18.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faa93fff640 1 -- 172.21.15.77:0/2712244637 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c003b80 con 0x7faaac102300 2024-03-16T09:39:18.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.651+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaac19fea0 con 0x7faaac102300 2024-03-16T09:39:18.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.652+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaac0fd820 con 0x7faaac102300 2024-03-16T09:39:18.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.653+0000 7faa93fff640 1 -- 172.21.15.77:0/2712244637 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7faa9c046020 con 0x7faaac102300 2024-03-16T09:39:18.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.654+0000 7faa93fff640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa940a2800 0x7faa940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.654+0000 7faaaa59c640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa940a2800 0x7faa940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.654+0000 7faa93fff640 1 -- 172.21.15.77:0/2712244637 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faa9c0f0880 con 0x7faaac102300 2024-03-16T09:39:18.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.654+0000 7faaaa59c640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa940a2800 0x7faa940a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faaac198fb0 tx=0x7faaa0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:18.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.658+0000 7faa93fff640 1 -- 172.21.15.77:0/2712244637 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa9c04d020 con 0x7faaac102300 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: Saving service mds.all spec with placement smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l;count:12 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:18.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:18.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:18 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:18.739 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: Saving service mds.all spec with placement smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l;count:12 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:18.740 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:18 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:18.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.834+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faaac0623e0 con 0x7faaac102300 2024-03-16T09:39:18.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.835+0000 7faa93fff640 1 -- 172.21.15.77:0/2712244637 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18468 (secure 0 0 0) 0x7faa9c0bddc0 con 0x7faaac102300 2024-03-16T09:39:18.838 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:18.838 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":82,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:07.686492+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa940a2800 msgr2=0x7faa940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa940a2800 0x7faa940a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faaac198fb0 tx=0x7faaa0005f90 comp rx=0 tx=0).stop 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 msgr2=0x7faaac197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac197aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faa9c006010 tx=0x7faa9c005e40 comp rx=0 tx=0).stop 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 shutdown_connections 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa940a2800 0x7faa940a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaac107560 0x7faaac19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faaac104c30 0x7faaac197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 --2- 172.21.15.77:0/2712244637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaac102300 0x7faaac197aa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 >> 172.21.15.77:0/2712244637 conn(0x7faaac0fbd30 msgr2=0x7faaac104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:18.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 shutdown_connections 2024-03-16T09:39:18.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.837+0000 7faab117e640 1 -- 172.21.15.77:0/2712244637 wait complete. 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: Saving service mds.all spec with placement smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l;count:12 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:18.850 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:18.851 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:18.851 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:18 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:18.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:39:18.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:39:18.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- 172.21.15.77:0/2772460943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc102290 msgr2=0x7f17fc108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 --2- 172.21.15.77:0/2772460943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc102290 0x7f17fc108b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f17f4009ca0 tx=0x7f17f402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- 172.21.15.77:0/2772460943 shutdown_connections 2024-03-16T09:39:18.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 --2- 172.21.15.77:0/2772460943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc102290 0x7f17fc108b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 --2- 172.21.15.77:0/2772460943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17fc1018d0 0x7f17fc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 --2- 172.21.15.77:0/2772460943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17fc1006e0 0x7f17fc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- 172.21.15.77:0/2772460943 >> 172.21.15.77:0/2772460943 conn(0x7f17fc0fbc70 msgr2=0x7f17fc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- 172.21.15.77:0/2772460943 shutdown_connections 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- 172.21.15.77:0/2772460943 wait complete. 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 Processor -- start 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- start start 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17fc1006e0 0x7f17fc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc1018d0 0x7f17fc19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17fc102290 0x7f17fc19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc10c280 con 0x7f17fc1018d0 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc10c100 con 0x7f17fc1006e0 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.954+0000 7f18047c6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc10c400 con 0x7f17fc102290 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc1018d0 0x7f17fc19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc1018d0 0x7f17fc19c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56916/0 (socket says 172.21.15.77:56916) 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 -- 172.21.15.77:0/2332681278 learned_addr learned my addr 172.21.15.77:0/2332681278 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1802562640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17fc1006e0 0x7f17fc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1802d63640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17fc102290 0x7f17fc19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 -- 172.21.15.77:0/2332681278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17fc102290 msgr2=0x7f17fc19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17fc102290 0x7f17fc19edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 -- 172.21.15.77:0/2332681278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17fc1006e0 msgr2=0x7f17fc0694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17fc1006e0 0x7f17fc0694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 -- 172.21.15.77:0/2332681278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc19f700 con 0x7f17fc1018d0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1802d63640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17fc102290 0x7f17fc19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f1801d61640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc1018d0 0x7f17fc19c7d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17f800d950 tx=0x7f17f800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f17f37fe640 1 -- 172.21.15.77:0/2332681278 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8017070 con 0x7f17fc1018d0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17fc19f990 con 0x7f17fc1018d0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f17f37fe640 1 -- 172.21.15.77:0/2332681278 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17f800de80 con 0x7f17fc1018d0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f17f37fe640 1 -- 172.21.15.77:0/2332681278 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f800f620 con 0x7f17fc1018d0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.955+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17fc19ff60 con 0x7f17fc1018d0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.956+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17fc107ad0 con 0x7f17fc1018d0 2024-03-16T09:39:18.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.957+0000 7f17f37fe640 1 -- 172.21.15.77:0/2332681278 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f17f8009c30 con 0x7f17fc1018d0 2024-03-16T09:39:18.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.958+0000 7f17f37fe640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f17e80a2800 0x7f17e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:18.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.958+0000 7f17f37fe640 1 -- 172.21.15.77:0/2332681278 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f17f80ce190 con 0x7f17fc1018d0 2024-03-16T09:39:18.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.958+0000 7f1802562640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f17e80a2800 0x7f17e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:18.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.958+0000 7f1802562640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f17e80a2800 0x7f17e80a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f17fc101730 tx=0x7f17ec009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:18.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:18.961+0000 7f17f37fe640 1 -- 172.21.15.77:0/2332681278 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17f809b6d0 con 0x7f17fc1018d0 2024-03-16T09:39:19.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.134+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f17fc02d400 con 0x7f17fc1018d0 2024-03-16T09:39:19.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.134+0000 7f17f37fe640 1 -- 172.21.15.77:0/2332681278 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f17f809afd0 con 0x7f17fc1018d0 2024-03-16T09:39:19.138 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:39:19.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.136+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f17e80a2800 msgr2=0x7f17e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:19.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.136+0000 7f18047c6640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f17e80a2800 0x7f17e80a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f17fc101730 tx=0x7f17ec009110 comp rx=0 tx=0).stop 2024-03-16T09:39:19.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.136+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc1018d0 msgr2=0x7f17fc19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:19.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.136+0000 7f18047c6640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc1018d0 0x7f17fc19c7d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17f800d950 tx=0x7f17f800dd50 comp rx=0 tx=0).stop 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 shutdown_connections 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f17e80a2800 0x7f17e80a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17fc102290 0x7f17fc19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17fc1018d0 0x7f17fc19c7d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 --2- 172.21.15.77:0/2332681278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17fc1006e0 0x7f17fc0694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 >> 172.21.15.77:0/2332681278 conn(0x7f17fc0fbc70 msgr2=0x7f17fc104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 shutdown_connections 2024-03-16T09:39:19.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.137+0000 7f18047c6640 1 -- 172.21.15.77:0/2332681278 wait complete. 2024-03-16T09:39:19.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 -- 172.21.15.77:0/3044143412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec100780 msgr2=0x7f6dec100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 --2- 172.21.15.77:0/3044143412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec100780 0x7f6dec100ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009900 tx=0x7f6ddc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 -- 172.21.15.77:0/3044143412 shutdown_connections 2024-03-16T09:39:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 --2- 172.21.15.77:0/3044143412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec102330 0x7f6dec108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 --2- 172.21.15.77:0/3044143412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dec101970 0x7f6dec101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 --2- 172.21.15.77:0/3044143412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec100780 0x7f6dec100ba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 -- 172.21.15.77:0/3044143412 >> 172.21.15.77:0/3044143412 conn(0x7f6dec0fbd10 msgr2=0x7f6dec0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 -- 172.21.15.77:0/3044143412 shutdown_connections 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.249+0000 7f6df0adc640 1 -- 172.21.15.77:0/3044143412 wait complete. 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 Processor -- start 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 -- start start 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec100780 0x7f6dec197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec101970 0x7f6dec197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dec102330 0x7f6dec19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec10c230 con 0x7f6dec101970 2024-03-16T09:39:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec10c0b0 con 0x7f6dec100780 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6df0adc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec10c3b0 con 0x7f6dec102330 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec101970 0x7f6dec197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec101970 0x7f6dec197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56934/0 (socket says 172.21.15.77:56934) 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 -- 172.21.15.77:0/1364848189 learned_addr learned my addr 172.21.15.77:0/1364848189 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6dead9d640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dec102330 0x7f6dec19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6dea59c640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec100780 0x7f6dec197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 -- 172.21.15.77:0/1364848189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dec102330 msgr2=0x7f6dec19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dec102330 0x7f6dec19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 -- 172.21.15.77:0/1364848189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec100780 msgr2=0x7f6dec197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec100780 0x7f6dec197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6de3fff640 1 -- 172.21.15.77:0/1364848189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec19f6d0 con 0x7f6dec101970 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.250+0000 7f6dea59c640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec100780 0x7f6dec197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.251+0000 7f6de3fff640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec101970 0x7f6dec197ff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400d770 tx=0x7f6dd400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.251+0000 7f6de37fe640 1 -- 172.21.15.77:0/1364848189 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4017070 con 0x7f6dec101970 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.251+0000 7f6de37fe640 1 -- 172.21.15.77:0/1364848189 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dd40054d0 con 0x7f6dec101970 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.251+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec19f9c0 con 0x7f6dec101970 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.251+0000 7f6de37fe640 1 -- 172.21.15.77:0/1364848189 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4004cf0 con 0x7f6dec101970 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.251+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec19fec0 con 0x7f6dec101970 2024-03-16T09:39:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.252+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dec1a00f0 con 0x7f6dec101970 2024-03-16T09:39:19.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.253+0000 7f6de37fe640 1 -- 172.21.15.77:0/1364848189 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6dd4004e90 con 0x7f6dec101970 2024-03-16T09:39:19.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.254+0000 7f6de37fe640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6dd80a2800 0x7f6dd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:19.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.254+0000 7f6dea59c640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6dd80a2800 0x7f6dd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:19.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.254+0000 7f6de37fe640 1 -- 172.21.15.77:0/1364848189 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6dd40ce440 con 0x7f6dec101970 2024-03-16T09:39:19.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.254+0000 7f6dea59c640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6dd80a2800 0x7f6dd80a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0096c0 tx=0x7f6ddc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.257+0000 7f6de37fe640 1 -- 172.21.15.77:0/1364848189 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6dd409b980 con 0x7f6dec101970 2024-03-16T09:39:19.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.471+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dec1a02c0 con 0x7f6dec101970 2024-03-16T09:39:19.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.472+0000 7f6de37fe640 1 -- 172.21.15.77:0/1364848189 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f6dd409b280 con 0x7f6dec101970 2024-03-16T09:39:19.477 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:19.477 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 1 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6dd80a2800 msgr2=0x7f6dd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6dd80a2800 0x7f6dd80a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0096c0 tx=0x7f6ddc000bc0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec101970 msgr2=0x7f6dec197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec101970 0x7f6dec197ff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400d770 tx=0x7f6dd400dc30 comp rx=0 tx=0).stop 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 shutdown_connections 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6dd80a2800 0x7f6dd80a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dec102330 0x7f6dec19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec101970 0x7f6dec197ff0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 --2- 172.21.15.77:0/1364848189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec100780 0x7f6dec197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 >> 172.21.15.77:0/1364848189 conn(0x7f6dec0fbd10 msgr2=0x7f6dec069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 shutdown_connections 2024-03-16T09:39:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:19.474+0000 7f6df0adc640 1 -- 172.21.15.77:0/1364848189 wait complete. 2024-03-16T09:39:19.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:19 smithi077 ceph-mon[32171]: Deploying daemon mds.l on smithi129 2024-03-16T09:39:19.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:19 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2712244637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:19.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:19 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2332681278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:19.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:19 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1364848189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:19.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:19 smithi129 ceph-mon[35841]: Deploying daemon mds.l on smithi129 2024-03-16T09:39:19.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:19 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2712244637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:19.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:19 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2332681278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:19.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:19 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1364848189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:19.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:19 smithi119 ceph-mon[35632]: Deploying daemon mds.l on smithi129 2024-03-16T09:39:19.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:19 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2712244637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:19.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:19 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2332681278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:19.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:19 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1364848189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:20.482 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 systemd[1]: Starting Ceph mds.l for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:20.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:20 smithi129 ceph-mon[35841]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:20.734 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 podman[65839]: 2024-03-16 09:39:20.381475609 +0000 UTC m=+0.018231344 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:20.734 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 podman[65839]: 2024-03-16 09:39:20.487725167 +0000 UTC m=+0.124480894 container create 292d32117075fdcde0a4ca2ccfd71b162a1fd691fe6dfafa5305f64b088b4fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:39:20.734 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 podman[65839]: 2024-03-16 09:39:20.732087009 +0000 UTC m=+0.368842749 container init 292d32117075fdcde0a4ca2ccfd71b162a1fd691fe6dfafa5305f64b088b4fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-16T09:39:20.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:20 smithi119 ceph-mon[35632]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:21.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:20 smithi077 ceph-mon[32171]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:21.051 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 podman[65839]: 2024-03-16 09:39:20.734710921 +0000 UTC m=+0.371466648 container start 292d32117075fdcde0a4ca2ccfd71b162a1fd691fe6dfafa5305f64b088b4fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T09:39:21.051 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 ceph-mds[65858]: -- 172.21.15.129:0/2039296209 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561a7a373a40 con 0x561a7a346800 2024-03-16T09:39:21.051 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l[65854]: starting mds.l at 2024-03-16T09:39:21.051 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 bash[65839]: 292d32117075fdcde0a4ca2ccfd71b162a1fd691fe6dfafa5305f64b088b4fb7 2024-03-16T09:39:21.051 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:39:20 smithi129 systemd[1]: Started Ceph mds.l for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:21.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 -- 172.21.15.77:0/3889077160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81018f0 msgr2=0x7f30f8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 --2- 172.21.15.77:0/3889077160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81018f0 0x7f30f8101d70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009900 tx=0x7f30e402bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 -- 172.21.15.77:0/3889077160 shutdown_connections 2024-03-16T09:39:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 --2- 172.21.15.77:0/3889077160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f81022b0 0x7f30f8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 --2- 172.21.15.77:0/3889077160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81018f0 0x7f30f8101d70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 --2- 172.21.15.77:0/3889077160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30f8100700 0x7f30f8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 -- 172.21.15.77:0/3889077160 >> 172.21.15.77:0/3889077160 conn(0x7f30f80fbc70 msgr2=0x7f30f80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 -- 172.21.15.77:0/3889077160 shutdown_connections 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 -- 172.21.15.77:0/3889077160 wait complete. 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 Processor -- start 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 -- start start 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8100700 0x7f30f80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.600+0000 7f30fea7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30f81018f0 0x7f30f819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fea7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81022b0 0x7f30f819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fea7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810c230 con 0x7f30f81022b0 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fea7e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810c0b0 con 0x7f30f8100700 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fea7e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810c3b0 con 0x7f30f81018f0 2024-03-16T09:39:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81022b0 0x7f30f819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81022b0 0x7f30f819edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56948/0 (socket says 172.21.15.77:56948) 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 -- 172.21.15.77:0/2649796267 learned_addr learned my addr 172.21.15.77:0/2649796267 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fc81a640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8100700 0x7f30f80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30f7fff640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30f81018f0 0x7f30f819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 -- 172.21.15.77:0/2649796267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30f81018f0 msgr2=0x7f30f819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30f81018f0 0x7f30f819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 -- 172.21.15.77:0/2649796267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8100700 msgr2=0x7f30f80694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8100700 0x7f30f80694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 -- 172.21.15.77:0/2649796267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f819f6e0 con 0x7f30f81022b0 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30f7fff640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30f81018f0 0x7f30f819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fd01b640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81022b0 0x7f30f819edd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f30f000eb10 tx=0x7f30f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30f5ffb640 1 -- 172.21.15.77:0/2649796267 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0004b20 con 0x7f30f81022b0 2024-03-16T09:39:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30f5ffb640 1 -- 172.21.15.77:0/2649796267 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30f0004cc0 con 0x7f30f81022b0 2024-03-16T09:39:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f819f970 con 0x7f30f81022b0 2024-03-16T09:39:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.601+0000 7f30f5ffb640 1 -- 172.21.15.77:0/2649796267 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0010690 con 0x7f30f81022b0 2024-03-16T09:39:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.602+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f819fe70 con 0x7f30f81022b0 2024-03-16T09:39:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.602+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f80685d0 con 0x7f30f81022b0 2024-03-16T09:39:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.603+0000 7f30f5ffb640 1 -- 172.21.15.77:0/2649796267 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f30f0005020 con 0x7f30f81022b0 2024-03-16T09:39:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.604+0000 7f30f5ffb640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f30e80a2800 0x7f30e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.604+0000 7f30f5ffb640 1 -- 172.21.15.77:0/2649796267 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f30f00cebe0 con 0x7f30f81022b0 2024-03-16T09:39:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.604+0000 7f30fc81a640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f30e80a2800 0x7f30e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.604+0000 7f30fc81a640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f30e80a2800 0x7f30e80a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30f8101750 tx=0x7f30ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.608+0000 7f30f5ffb640 1 -- 172.21.15.77:0/2649796267 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30f009c120 con 0x7f30f81022b0 2024-03-16T09:39:21.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.821+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30f8107aa0 con 0x7f30f81022b0 2024-03-16T09:39:21.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.822+0000 7f30f5ffb640 1 -- 172.21.15.77:0/2649796267 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f30f009ba20 con 0x7f30f81022b0 2024-03-16T09:39:21.825 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:21.825 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T09:39:21.825 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 2 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.823+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f30e80a2800 msgr2=0x7f30e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.823+0000 7f30fea7e640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f30e80a2800 0x7f30e80a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30f8101750 tx=0x7f30ec005f90 comp rx=0 tx=0).stop 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81022b0 msgr2=0x7f30f819edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81022b0 0x7f30f819edd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f30f000eb10 tx=0x7f30f000c2d0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 shutdown_connections 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f30e80a2800 0x7f30e80a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f81022b0 0x7f30f819edd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30f81018f0 0x7f30f819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 --2- 172.21.15.77:0/2649796267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8100700 0x7f30f80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 >> 172.21.15.77:0/2649796267 conn(0x7f30f80fbc70 msgr2=0x7f30f8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 shutdown_connections 2024-03-16T09:39:21.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:21.824+0000 7f30fea7e640 1 -- 172.21.15.77:0/2649796267 wait complete. 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6832/3738546284,v1:172.21.15.129:6833/3738546284] up:boot 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: fsmap 1 up:standby 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:22.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:21 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6832/3738546284,v1:172.21.15.129:6833/3738546284] up:boot 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: fsmap 1 up:standby 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:22.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:22.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:21 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:22.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6832/3738546284,v1:172.21.15.129:6833/3738546284] up:boot 2024-03-16T09:39:22.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: fsmap 1 up:standby 2024-03-16T09:39:22.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T09:39:22.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:22.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:22.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:22.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:21 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:22.895 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:22 smithi119 ceph-mon[35632]: Deploying daemon mds.k on smithi119 2024-03-16T09:39:22.896 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:22 smithi119 ceph-mon[35632]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:22.896 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:22 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2649796267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:23.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:22 smithi129 ceph-mon[35841]: Deploying daemon mds.k on smithi119 2024-03-16T09:39:23.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:22 smithi129 ceph-mon[35841]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:23.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:22 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2649796267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:23.190 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 systemd[1]: Starting Ceph mds.k for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:23.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:22 smithi077 ceph-mon[32171]: Deploying daemon mds.k on smithi119 2024-03-16T09:39:23.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:22 smithi077 ceph-mon[32171]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:23.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:22 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2649796267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:23.551 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 podman[65077]: 2024-03-16 09:39:23.188735431 +0000 UTC m=+0.018001141 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:23.554 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 podman[65077]: 2024-03-16 09:39:23.286811144 +0000 UTC m=+0.116076854 container create 7002a90ce45b5e722ee9d2ebc0396cf5e81e7f0ef0449220c5dafb0c309749f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-16T09:39:23.554 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 podman[65077]: 2024-03-16 09:39:23.522960238 +0000 UTC m=+0.352225941 container init 7002a90ce45b5e722ee9d2ebc0396cf5e81e7f0ef0449220c5dafb0c309749f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-16T09:39:23.554 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 podman[65077]: 2024-03-16 09:39:23.525746485 +0000 UTC m=+0.355012196 container start 7002a90ce45b5e722ee9d2ebc0396cf5e81e7f0ef0449220c5dafb0c309749f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T09:39:23.838 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:23.940 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 ceph-mds[65096]: -- 172.21.15.119:0/2418185585 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56136dc3da40 con 0x56136dc10c00 2024-03-16T09:39:23.940 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k[65092]: starting mds.k at 2024-03-16T09:39:23.940 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 bash[65077]: 7002a90ce45b5e722ee9d2ebc0396cf5e81e7f0ef0449220c5dafb0c309749f2 2024-03-16T09:39:23.940 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:39:23 smithi119 systemd[1]: Started Ceph mds.k for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:23.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2577639494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 msgr2=0x7fc0dc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:23.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2577639494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc108b40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009ca0 tx=0x7fc0d402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:23.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2577639494 shutdown_connections 2024-03-16T09:39:23.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2577639494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc108b40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:23.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2577639494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0dc1018f0 0x7fc0dc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:23.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2577639494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0dc100700 0x7fc0dc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:23.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2577639494 >> 172.21.15.77:0/2577639494 conn(0x7fc0dc0fbc70 msgr2=0x7fc0dc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:23.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.939+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2577639494 shutdown_connections 2024-03-16T09:39:23.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.940+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2577639494 wait complete. 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.940+0000 7fc0e0d40640 1 Processor -- start 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.940+0000 7fc0e0d40640 1 -- start start 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0e0d40640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0dc100700 0x7fc0dc1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0e0d40640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0dc1018f0 0x7fc0dc196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0e0d40640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0e0d40640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc10c230 con 0x7fc0dc1022b0 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0e0d40640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc10c0b0 con 0x7fc0dc100700 2024-03-16T09:39:23.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0e0d40640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc10c3b0 con 0x7fc0dc1018f0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56964/0 (socket says 172.21.15.77:56964) 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 -- 172.21.15.77:0/2265281452 learned_addr learned my addr 172.21.15.77:0/2265281452 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0da59c640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0dc100700 0x7fc0dc1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0d9d9b640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0dc1018f0 0x7fc0dc196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 -- 172.21.15.77:0/2265281452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0dc1018f0 msgr2=0x7fc0dc196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0dc1018f0 0x7fc0dc196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 -- 172.21.15.77:0/2265281452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0dc100700 msgr2=0x7fc0dc1968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0dc100700 0x7fc0dc1968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 -- 172.21.15.77:0/2265281452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc19e4f0 con 0x7fc0dc1022b0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0d9d9b640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0dc1018f0 0x7fc0dc196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0dad9d640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc19de80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc0d402c700 tx=0x7fc0d40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.941+0000 7fc0cb7fe640 1 -- 172.21.15.77:0/2265281452 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4041020 con 0x7fc0dc1022b0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.942+0000 7fc0cb7fe640 1 -- 172.21.15.77:0/2265281452 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0d4003e20 con 0x7fc0dc1022b0 2024-03-16T09:39:23.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.942+0000 7fc0cb7fe640 1 -- 172.21.15.77:0/2265281452 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4039040 con 0x7fc0dc1022b0 2024-03-16T09:39:23.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.942+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc19e780 con 0x7fc0dc1022b0 2024-03-16T09:39:23.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.942+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc19ecc0 con 0x7fc0dc1022b0 2024-03-16T09:39:23.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.943+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0dc107aa0 con 0x7fc0dc1022b0 2024-03-16T09:39:23.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.944+0000 7fc0cb7fe640 1 -- 172.21.15.77:0/2265281452 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc0d4044020 con 0x7fc0dc1022b0 2024-03-16T09:39:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.944+0000 7fc0cb7fe640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0c40a2880 0x7fc0c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.945+0000 7fc0da59c640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0c40a2880 0x7fc0c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.945+0000 7fc0cb7fe640 1 -- 172.21.15.77:0/2265281452 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc0d40f0c90 con 0x7fc0dc1022b0 2024-03-16T09:39:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.945+0000 7fc0da59c640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0c40a2880 0x7fc0c40a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc005fd0 tx=0x7fc0cc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:23.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:23.948+0000 7fc0cb7fe640 1 -- 172.21.15.77:0/2265281452 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc0d404b020 con 0x7fc0dc1022b0 2024-03-16T09:39:24.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.170+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0dc02d370 con 0x7fc0dc1022b0 2024-03-16T09:39:24.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.171+0000 7fc0cb7fe640 1 -- 172.21.15.77:0/2265281452 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fc0dc02d370 con 0x7fc0dc1022b0 2024-03-16T09:39:24.174 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:24.174 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-16T09:39:24.174 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0c40a2880 msgr2=0x7fc0c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0c40a2880 0x7fc0c40a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc005fd0 tx=0x7fc0cc00b390 comp rx=0 tx=0).stop 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 msgr2=0x7fc0dc19de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc19de80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc0d402c700 tx=0x7fc0d40328a0 comp rx=0 tx=0).stop 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 shutdown_connections 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc0c40a2880 0x7fc0c40a4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0dc1022b0 0x7fc0dc19de80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0dc1018f0 0x7fc0dc196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 --2- 172.21.15.77:0/2265281452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0dc100700 0x7fc0dc1968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 >> 172.21.15.77:0/2265281452 conn(0x7fc0dc0fbc70 msgr2=0x7fc0dc104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 shutdown_connections 2024-03-16T09:39:24.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:24.173+0000 7fc0e0d40640 1 -- 172.21.15.77:0/2265281452 wait complete. 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: Deploying daemon mds.j on smithi077 2024-03-16T09:39:24.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:24.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6832/3136672591,v1:172.21.15.119:6833/3136672591] up:boot 2024-03-16T09:39:24.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: fsmap 2 up:standby 2024-03-16T09:39:24.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T09:39:24.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:24 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2265281452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:25.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:25.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:25.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:25.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: Deploying daemon mds.j on smithi077 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6832/3136672591,v1:172.21.15.119:6833/3136672591] up:boot 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: fsmap 2 up:standby 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T09:39:25.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:24 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2265281452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:25.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:25.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: Deploying daemon mds.j on smithi077 2024-03-16T09:39:25.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:25.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6832/3136672591,v1:172.21.15.119:6833/3136672591] up:boot 2024-03-16T09:39:25.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: fsmap 2 up:standby 2024-03-16T09:39:25.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T09:39:25.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:24 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2265281452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:25.759 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:25 smithi077 systemd[1]: Starting Ceph mds.j for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:26.165 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:25 smithi077 podman[72952]: 2024-03-16 09:39:25.91081629 +0000 UTC m=+0.100596650 container create 20aa64164ac26d48e5d129b7a07092f5e1dea3cbb116429839d1d68567ed349f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T09:39:26.166 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:25 smithi077 podman[72952]: 2024-03-16 09:39:25.828610881 +0000 UTC m=+0.018391244 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:26.166 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:26 smithi077 podman[72952]: 2024-03-16 09:39:26.163475912 +0000 UTC m=+0.353256280 container init 20aa64164ac26d48e5d129b7a07092f5e1dea3cbb116429839d1d68567ed349f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:39:26.189 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:26.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 -- 172.21.15.77:0/1107852403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 msgr2=0x7f0e08107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:26.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 --2- 172.21.15.77:0/1107852403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08107000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009900 tx=0x7f0df802bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:26.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 -- 172.21.15.77:0/1107852403 shutdown_connections 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 --2- 172.21.15.77:0/1107852403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e08107540 0x7f0e08109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 --2- 172.21.15.77:0/1107852403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08107000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 --2- 172.21.15.77:0/1107852403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e081022e0 0x7f0e081046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 -- 172.21.15.77:0/1107852403 >> 172.21.15.77:0/1107852403 conn(0x7f0e080fbc90 msgr2=0x7f0e080fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.298+0000 7f0e1024b640 1 -- 172.21.15.77:0/1107852403 shutdown_connections 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 -- 172.21.15.77:0/1107852403 wait complete. 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 Processor -- start 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 -- start start 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e081022e0 0x7f0e08197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e08107540 0x7f0e0819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0810d350 con 0x7f0e08104c10 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0810d1d0 con 0x7f0e08107540 2024-03-16T09:39:26.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e1024b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0810d4d0 con 0x7f0e081022e0 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e0d7e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e0d7e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54848/0 (socket says 172.21.15.77:54848) 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e0d7e6640 1 -- 172.21.15.77:0/767303586 learned_addr learned my addr 172.21.15.77:0/767303586 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e0dfe7640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e081022e0 0x7f0e08197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.299+0000 7f0e0e7e8640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e08107540 0x7f0e0819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.300+0000 7f0e0d7e6640 1 -- 172.21.15.77:0/767303586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e081022e0 msgr2=0x7f0e08197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.300+0000 7f0e0d7e6640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e081022e0 0x7f0e08197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.300+0000 7f0e0d7e6640 1 -- 172.21.15.77:0/767303586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e08107540 msgr2=0x7f0e0819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.300+0000 7f0e0d7e6640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e08107540 0x7f0e0819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.300+0000 7f0e0d7e6640 1 -- 172.21.15.77:0/767303586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0819f720 con 0x7f0e08104c10 2024-03-16T09:39:26.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.300+0000 7f0e0d7e6640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08198040 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0df8000c00 tx=0x7f0df8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.301+0000 7f0dfeffd640 1 -- 172.21.15.77:0/767303586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8043020 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.301+0000 7f0dfeffd640 1 -- 172.21.15.77:0/767303586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0df8004440 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.301+0000 7f0dfeffd640 1 -- 172.21.15.77:0/767303586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8003c60 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.301+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0819f9b0 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.301+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0819ff00 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.302+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e080fd780 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.303+0000 7f0dfeffd640 1 -- 172.21.15.77:0/767303586 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0df8046020 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.304+0000 7f0dfeffd640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0df40a2800 0x7f0df40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.304+0000 7f0e0dfe7640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0df40a2800 0x7f0df40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.304+0000 7f0dfeffd640 1 -- 172.21.15.77:0/767303586 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0df80f07e0 con 0x7f0e08104c10 2024-03-16T09:39:26.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.304+0000 7f0e0dfe7640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0df40a2800 0x7f0df40a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005d00 tx=0x7f0e0400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:26.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.311+0000 7f0dfeffd640 1 -- 172.21.15.77:0/767303586 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0df804d020 con 0x7f0e08104c10 2024-03-16T09:39:26.509 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:26 smithi077 podman[72952]: 2024-03-16 09:39:26.166332565 +0000 UTC m=+0.356112930 container start 20aa64164ac26d48e5d129b7a07092f5e1dea3cbb116429839d1d68567ed349f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-16T09:39:26.509 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mds[72971]: -- 172.21.15.77:0/2040136366 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564497b17a40 con 0x564497aea800 2024-03-16T09:39:26.509 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j[72967]: starting mds.j at 2024-03-16T09:39:26.509 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:26 smithi077 bash[72952]: 20aa64164ac26d48e5d129b7a07092f5e1dea3cbb116429839d1d68567ed349f 2024-03-16T09:39:26.509 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:39:26 smithi077 systemd[1]: Started Ceph mds.j for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.530+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e0802d400 con 0x7f0e08104c10 2024-03-16T09:39:26.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.531+0000 7f0dfeffd640 1 -- 172.21.15.77:0/767303586 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f0df80bdd20 con 0x7f0e08104c10 2024-03-16T09:39:26.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:26.534 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-16T09:39:26.534 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2024-03-16T09:39:26.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0df40a2800 msgr2=0x7f0df40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:26.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0df40a2800 0x7f0df40a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005d00 tx=0x7f0e0400b410 comp rx=0 tx=0).stop 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 msgr2=0x7f0e08198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08198040 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0df8000c00 tx=0x7f0df8004040 comp rx=0 tx=0).stop 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 shutdown_connections 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0df40a2800 0x7f0df40a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e08107540 0x7f0e0819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08104c10 0x7f0e08198040 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 --2- 172.21.15.77:0/767303586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e081022e0 0x7f0e08197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 >> 172.21.15.77:0/767303586 conn(0x7f0e080fbc90 msgr2=0x7f0e08102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 shutdown_connections 2024-03-16T09:39:26.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:26.533+0000 7f0e1024b640 1 -- 172.21.15.77:0/767303586 wait complete. 2024-03-16T09:39:27.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:27.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.079 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.080 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:27.080 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:27.080 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:27.080 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:26 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/767303586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:27.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:26 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/767303586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:26 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/767303586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:28.185 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:27 smithi129 ceph-mon[35841]: Deploying daemon mds.d on smithi077 2024-03-16T09:39:28.185 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:27 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.77:6834/1282436458,v1:172.21.15.77:6835/1282436458] up:boot 2024-03-16T09:39:28.185 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:27 smithi129 ceph-mon[35841]: fsmap 3 up:standby 2024-03-16T09:39:28.185 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:27 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T09:39:28.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:27 smithi119 ceph-mon[35632]: Deploying daemon mds.d on smithi077 2024-03-16T09:39:28.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:27 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.77:6834/1282436458,v1:172.21.15.77:6835/1282436458] up:boot 2024-03-16T09:39:28.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:27 smithi119 ceph-mon[35632]: fsmap 3 up:standby 2024-03-16T09:39:28.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:27 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T09:39:28.219 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:27 smithi077 ceph-mon[32171]: Deploying daemon mds.d on smithi077 2024-03-16T09:39:28.219 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:27 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.77:6834/1282436458,v1:172.21.15.77:6835/1282436458] up:boot 2024-03-16T09:39:28.219 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:27 smithi077 ceph-mon[32171]: fsmap 3 up:standby 2024-03-16T09:39:28.219 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:27 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T09:39:28.509 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 systemd[1]: Starting Ceph mds.d for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:28.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 -- 172.21.15.77:0/3198952584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a81033e0 msgr2=0x7fc5a8109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 --2- 172.21.15.77:0/3198952584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a81033e0 0x7fc5a8109c70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0009ca0 tx=0x7fc5a002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 -- 172.21.15.77:0/3198952584 shutdown_connections 2024-03-16T09:39:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 --2- 172.21.15.77:0/3198952584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a81033e0 0x7fc5a8109c70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 --2- 172.21.15.77:0/3198952584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a8102a20 0x7fc5a8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 --2- 172.21.15.77:0/3198952584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a8101830 0x7fc5a8101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 -- 172.21.15.77:0/3198952584 >> 172.21.15.77:0/3198952584 conn(0x7fc5a80fcd80 msgr2=0x7fc5a80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 -- 172.21.15.77:0/3198952584 shutdown_connections 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 -- 172.21.15.77:0/3198952584 wait complete. 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 Processor -- start 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.654+0000 7fc5adac0640 1 -- start start 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8101830 0x7fc5a8075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a8102a20 0x7fc5a8073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81033e0 0x7fc5a8074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a810d340 con 0x7fc5a8101830 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a810d1c0 con 0x7fc5a81033e0 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a810d4c0 con 0x7fc5a8102a20 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8101830 0x7fc5a8075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8101830 0x7fc5a8075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54860/0 (socket says 172.21.15.77:54860) 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 -- 172.21.15.77:0/1831138326 learned_addr learned my addr 172.21.15.77:0/1831138326 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:28.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a7fff640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81033e0 0x7fc5a8074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a6ffd640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a8102a20 0x7fc5a8073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 -- 172.21.15.77:0/1831138326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a8102a20 msgr2=0x7fc5a8073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a8102a20 0x7fc5a8073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 -- 172.21.15.77:0/1831138326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81033e0 msgr2=0x7fc5a8074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81033e0 0x7fc5a8074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 -- 172.21.15.77:0/1831138326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a80749a0 con 0x7fc5a8101830 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a77fe640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8101830 0x7fc5a8075590 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc59800d950 tx=0x7fc59800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a4ff9640 1 -- 172.21.15.77:0/1831138326 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598017020 con 0x7fc5a8101830 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a4ff9640 1 -- 172.21.15.77:0/1831138326 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc598009c30 con 0x7fc5a8101830 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a81a1b10 con 0x7fc5a8101830 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5a4ff9640 1 -- 172.21.15.77:0/1831138326 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59800f6b0 con 0x7fc5a8101830 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.655+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a81a2060 con 0x7fc5a8101830 2024-03-16T09:39:28.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.656+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a8108b50 con 0x7fc5a8101830 2024-03-16T09:39:28.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.657+0000 7fc5a4ff9640 1 -- 172.21.15.77:0/1831138326 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc59801a020 con 0x7fc5a8101830 2024-03-16T09:39:28.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.658+0000 7fc5a4ff9640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2800 0x7fc5900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:28.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.658+0000 7fc5a6ffd640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2800 0x7fc5900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:28.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.658+0000 7fc5a4ff9640 1 -- 172.21.15.77:0/1831138326 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc5980ce3b0 con 0x7fc5a8101830 2024-03-16T09:39:28.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.658+0000 7fc5a6ffd640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2800 0x7fc5900a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8075300 tx=0x7fc59c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:28.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.662+0000 7fc5a4ff9640 1 -- 172.21.15.77:0/1831138326 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc598009de0 con 0x7fc5a8101830 2024-03-16T09:39:28.830 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 podman[73300]: 2024-03-16 09:39:28.510090403 +0000 UTC m=+0.018011537 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:28.831 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 podman[73300]: 2024-03-16 09:39:28.631064605 +0000 UTC m=+0.138985715 container create d6b5984fc6ae12664bb522de7d106b974d622f6cfca11301868dfde466c125d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T09:39:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.896+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5a802d1b0 con 0x7fc5a8101830 2024-03-16T09:39:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.897+0000 7fc5a4ff9640 1 -- 172.21.15.77:0/1831138326 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fc59809b8f0 con 0x7fc5a8101830 2024-03-16T09:39:28.900 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:28.900 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-16T09:39:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 4 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2800 msgr2=0x7fc5900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2800 0x7fc5900a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8075300 tx=0x7fc59c005f90 comp rx=0 tx=0).stop 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8101830 msgr2=0x7fc5a8075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8101830 0x7fc5a8075590 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc59800d950 tx=0x7fc59800de10 comp rx=0 tx=0).stop 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 shutdown_connections 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2800 0x7fc5900a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81033e0 0x7fc5a8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a8102a20 0x7fc5a8073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 --2- 172.21.15.77:0/1831138326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8101830 0x7fc5a8075590 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.899+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 >> 172.21.15.77:0/1831138326 conn(0x7fc5a80fcd80 msgr2=0x7fc5a8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:28.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.900+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 shutdown_connections 2024-03-16T09:39:28.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:28.900+0000 7fc5adac0640 1 -- 172.21.15.77:0/1831138326 wait complete. 2024-03-16T09:39:29.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:28 smithi119 ceph-mon[35632]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:28 smithi077 ceph-mon[32171]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:29.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 podman[73300]: 2024-03-16 09:39:28.900657822 +0000 UTC m=+0.408578956 container init d6b5984fc6ae12664bb522de7d106b974d622f6cfca11301868dfde466c125d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-03-16T09:39:29.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 podman[73300]: 2024-03-16 09:39:28.903499039 +0000 UTC m=+0.411420156 container start d6b5984fc6ae12664bb522de7d106b974d622f6cfca11301868dfde466c125d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T09:39:29.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 ceph-mds[73360]: -- 172.21.15.77:0/3460465433 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563840d9fa40 con 0x563840d72c00 2024-03-16T09:39:29.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d[73354]: starting mds.d at 2024-03-16T09:39:29.260 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 bash[73300]: d6b5984fc6ae12664bb522de7d106b974d622f6cfca11301868dfde466c125d0 2024-03-16T09:39:29.260 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:39:28 smithi077 systemd[1]: Started Ceph mds.d for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:28 smithi129 ceph-mon[35841]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:29.995 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:29 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1831138326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:29.995 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:29 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:29.995 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:29 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:29.995 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:29 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:29.995 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:29 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:29.995 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:29 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:29.995 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:29 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:29 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1831138326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:29 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:29 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:29 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:30.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:29 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:29 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:30.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:29 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:29 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1831138326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:29 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:29 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:29 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:29 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:29 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:29 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:30.915 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:30.933 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:30 smithi119 ceph-mon[35632]: Deploying daemon mds.e on smithi119 2024-03-16T09:39:30.933 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:30 smithi119 ceph-mon[35632]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:30.933 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:30 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.77:6836/3058701505,v1:172.21.15.77:6837/3058701505] up:boot 2024-03-16T09:39:30.933 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:30 smithi119 ceph-mon[35632]: fsmap 4 up:standby 2024-03-16T09:39:30.933 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:30 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T09:39:31.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 -- 172.21.15.77:0/3064529316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3944102a60 msgr2=0x7f3944104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 --2- 172.21.15.77:0/3064529316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3944102a60 0x7f3944104ed0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3934009900 tx=0x7f393402bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 -- 172.21.15.77:0/3064529316 shutdown_connections 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 --2- 172.21.15.77:0/3064529316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3944105410 0x7f3944105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 --2- 172.21.15.77:0/3064529316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3944102a60 0x7f3944104ed0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 --2- 172.21.15.77:0/3064529316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39441000f0 0x7f3944102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 -- 172.21.15.77:0/3064529316 >> 172.21.15.77:0/3064529316 conn(0x7f39440fbd10 msgr2=0x7f39440fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 -- 172.21.15.77:0/3064529316 shutdown_connections 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.025+0000 7f394bf3b640 1 -- 172.21.15.77:0/3064529316 wait complete. 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 Processor -- start 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 -- start start 2024-03-16T09:39:31.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39441000f0 0x7f394419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3944102a60 0x7f394419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3944105410 0x7f39441a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3944113910 con 0x7f39441000f0 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3944113790 con 0x7f3944105410 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394bf3b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3944113a90 con 0x7f3944102a60 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39441000f0 0x7f394419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39441000f0 0x7f394419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54896/0 (socket says 172.21.15.77:54896) 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 -- 172.21.15.77:0/982089648 learned_addr learned my addr 172.21.15.77:0/982089648 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f394a4d8640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3944105410 0x7f39441a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 -- 172.21.15.77:0/982089648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3944102a60 msgr2=0x7f394419e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3944102a60 0x7f394419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 -- 172.21.15.77:0/982089648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3944105410 msgr2=0x7f39441a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:31.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3944105410 0x7f39441a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.026+0000 7f3949cd7640 1 -- 172.21.15.77:0/982089648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39441a5c40 con 0x7f39441000f0 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.027+0000 7f3949cd7640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39441000f0 0x7f394419e020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f394000e9f0 tx=0x7f394000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.027+0000 7f393affd640 1 -- 172.21.15.77:0/982089648 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394000cd80 con 0x7f39441000f0 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.027+0000 7f393affd640 1 -- 172.21.15.77:0/982089648 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39400108e0 con 0x7f39441000f0 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.027+0000 7f393affd640 1 -- 172.21.15.77:0/982089648 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940018920 con 0x7f39441000f0 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.027+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39441a5f30 con 0x7f39441000f0 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.027+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39441a63f0 con 0x7f39441000f0 2024-03-16T09:39:31.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.028+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39440fd800 con 0x7f39441000f0 2024-03-16T09:39:31.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.029+0000 7f393affd640 1 -- 172.21.15.77:0/982089648 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3940010a90 con 0x7f39441000f0 2024-03-16T09:39:31.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.030+0000 7f393affd640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39300a2800 0x7f39300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:31.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.030+0000 7f39494d6640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39300a2800 0x7f39300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:31.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.030+0000 7f393affd640 1 -- 172.21.15.77:0/982089648 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f39400ce980 con 0x7f39441000f0 2024-03-16T09:39:31.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.030+0000 7f39494d6640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39300a2800 0x7f39300a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3934036040 tx=0x7f3934000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:31.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.034+0000 7f393affd640 1 -- 172.21.15.77:0/982089648 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f394009bec0 con 0x7f39441000f0 2024-03-16T09:39:31.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 systemd[1]: Starting Ceph mds.e for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:31.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.247+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f394402d1b0 con 0x7f39441000f0 2024-03-16T09:39:31.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.247+0000 7f393affd640 1 -- 172.21.15.77:0/982089648 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f394009b7c0 con 0x7f39441000f0 2024-03-16T09:39:31.250 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:31.250 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-16T09:39:31.251 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 5 2024-03-16T09:39:31.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39300a2800 msgr2=0x7f39300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39300a2800 0x7f39300a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3934036040 tx=0x7f3934000bc0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39441000f0 msgr2=0x7f394419e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39441000f0 0x7f394419e020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f394000e9f0 tx=0x7f394000eeb0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 shutdown_connections 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39300a2800 0x7f39300a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3944105410 0x7f39441a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3944102a60 0x7f394419e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 --2- 172.21.15.77:0/982089648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39441000f0 0x7f394419e020 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.249+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 >> 172.21.15.77:0/982089648 conn(0x7f39440fbd10 msgr2=0x7f3944100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.250+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 shutdown_connections 2024-03-16T09:39:31.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:31.250+0000 7f394bf3b640 1 -- 172.21.15.77:0/982089648 wait complete. 2024-03-16T09:39:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:30 smithi077 ceph-mon[32171]: Deploying daemon mds.e on smithi119 2024-03-16T09:39:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:30 smithi077 ceph-mon[32171]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:30 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.77:6836/3058701505,v1:172.21.15.77:6837/3058701505] up:boot 2024-03-16T09:39:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:30 smithi077 ceph-mon[32171]: fsmap 4 up:standby 2024-03-16T09:39:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:30 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T09:39:31.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:30 smithi129 ceph-mon[35841]: Deploying daemon mds.e on smithi119 2024-03-16T09:39:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:30 smithi129 ceph-mon[35841]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:30 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.77:6836/3058701505,v1:172.21.15.77:6837/3058701505] up:boot 2024-03-16T09:39:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:30 smithi129 ceph-mon[35841]: fsmap 4 up:standby 2024-03-16T09:39:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:30 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T09:39:31.595 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 podman[65399]: 2024-03-16 09:39:31.236729211 +0000 UTC m=+0.018436702 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:31.595 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 podman[65399]: 2024-03-16 09:39:31.357150696 +0000 UTC m=+0.138858225 container create 99588bee2562e8bd378e9f5179327c2b9aee61681c966083e143ae128a7bb1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T09:39:31.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/982089648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:31.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:31.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:31.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:31.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:31.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:31.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:31.941 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 podman[65399]: 2024-03-16 09:39:31.593255997 +0000 UTC m=+0.374963499 container init 99588bee2562e8bd378e9f5179327c2b9aee61681c966083e143ae128a7bb1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:39:31.941 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 podman[65399]: 2024-03-16 09:39:31.595846629 +0000 UTC m=+0.377554121 container start 99588bee2562e8bd378e9f5179327c2b9aee61681c966083e143ae128a7bb1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-16T09:39:31.941 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-mds[65419]: -- 172.21.15.119:0/4018495703 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa05ae7a40 con 0x55fa05abb000 2024-03-16T09:39:31.941 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e[65415]: starting mds.e at 2024-03-16T09:39:31.941 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 bash[65399]: 99588bee2562e8bd378e9f5179327c2b9aee61681c966083e143ae128a7bb1b5 2024-03-16T09:39:31.941 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:39:31 smithi119 systemd[1]: Started Ceph mds.e for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:31 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/982089648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:31 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:31 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:31 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:31 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:31 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:31 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:32.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:31 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/982089648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:32.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:31 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:32.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:31 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:32.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:31 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:32.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:31 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:32.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:31 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:32.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:31 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:33.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:32 smithi129 ceph-mon[35841]: Deploying daemon mds.f on smithi129 2024-03-16T09:39:33.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:32 smithi129 ceph-mon[35841]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:33.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:32 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6834/296042620,v1:172.21.15.119:6835/296042620] up:boot 2024-03-16T09:39:33.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:32 smithi129 ceph-mon[35841]: fsmap 5 up:standby 2024-03-16T09:39:33.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:32 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T09:39:33.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:32 smithi119 ceph-mon[35632]: Deploying daemon mds.f on smithi129 2024-03-16T09:39:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:32 smithi119 ceph-mon[35632]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:32 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6834/296042620,v1:172.21.15.119:6835/296042620] up:boot 2024-03-16T09:39:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:32 smithi119 ceph-mon[35632]: fsmap 5 up:standby 2024-03-16T09:39:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:32 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T09:39:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:32 smithi077 ceph-mon[32171]: Deploying daemon mds.f on smithi129 2024-03-16T09:39:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:32 smithi077 ceph-mon[32171]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:32 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6834/296042620,v1:172.21.15.119:6835/296042620] up:boot 2024-03-16T09:39:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:32 smithi077 ceph-mon[32171]: fsmap 5 up:standby 2024-03-16T09:39:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:32 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T09:39:33.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:33.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 -- 172.21.15.77:0/1147536913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd308100700 msgr2=0x7fd308100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:33.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 --2- 172.21.15.77:0/1147536913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd308100700 0x7fd308100b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009900 tx=0x7fd2f802bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 -- 172.21.15.77:0/1147536913 shutdown_connections 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 --2- 172.21.15.77:0/1147536913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3081022b0 0x7fd308108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 --2- 172.21.15.77:0/1147536913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3081018f0 0x7fd308101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 --2- 172.21.15.77:0/1147536913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd308100700 0x7fd308100b20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 -- 172.21.15.77:0/1147536913 >> 172.21.15.77:0/1147536913 conn(0x7fd3080fbc70 msgr2=0x7fd3080fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 -- 172.21.15.77:0/1147536913 shutdown_connections 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.373+0000 7fd31020b640 1 -- 172.21.15.77:0/1147536913 wait complete. 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 Processor -- start 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 -- start start 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd308100700 0x7fd3081979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3081018f0 0x7fd308197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3081022b0 0x7fd30819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:33.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810c200 con 0x7fd3081022b0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810c080 con 0x7fd308100700 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd31020b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810c380 con 0x7fd3081018f0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3081022b0 0x7fd30819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3081022b0 0x7fd30819ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54924/0 (socket says 172.21.15.77:54924) 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 -- 172.21.15.77:0/2935284106 learned_addr learned my addr 172.21.15.77:0/2935284106 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30d7a6640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3081018f0 0x7fd308197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30dfa7640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd308100700 0x7fd3081979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 -- 172.21.15.77:0/2935284106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3081018f0 msgr2=0x7fd308197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3081018f0 0x7fd308197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 -- 172.21.15.77:0/2935284106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd308100700 msgr2=0x7fd3081979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd308100700 0x7fd3081979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.374+0000 7fd30e7a8640 1 -- 172.21.15.77:0/2935284106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30819f5f0 con 0x7fd3081022b0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.375+0000 7fd30e7a8640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3081022b0 0x7fd30819ef80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd30000eaa0 tx=0x7fd30000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.375+0000 7fd2feffd640 1 -- 172.21.15.77:0/2935284106 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300004d20 con 0x7fd3081022b0 2024-03-16T09:39:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.375+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd30819f8e0 con 0x7fd3081022b0 2024-03-16T09:39:33.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.375+0000 7fd2feffd640 1 -- 172.21.15.77:0/2935284106 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd300005020 con 0x7fd3081022b0 2024-03-16T09:39:33.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.375+0000 7fd2feffd640 1 -- 172.21.15.77:0/2935284106 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000116b0 con 0x7fd3081022b0 2024-03-16T09:39:33.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.375+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd30819fde0 con 0x7fd3081022b0 2024-03-16T09:39:33.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.376+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3081a0040 con 0x7fd3081022b0 2024-03-16T09:39:33.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.377+0000 7fd2feffd640 1 -- 172.21.15.77:0/2935284106 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd3000051d0 con 0x7fd3081022b0 2024-03-16T09:39:33.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.378+0000 7fd2feffd640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2f40a2800 0x7fd2f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:33.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.378+0000 7fd30dfa7640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2f40a2800 0x7fd2f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:33.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.378+0000 7fd2feffd640 1 -- 172.21.15.77:0/2935284106 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd3000cec40 con 0x7fd3081022b0 2024-03-16T09:39:33.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.378+0000 7fd30dfa7640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2f40a2800 0x7fd2f40a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8006010 tx=0x7fd2f8005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:33.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.381+0000 7fd2feffd640 1 -- 172.21.15.77:0/2935284106 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd300016030 con 0x7fd3081022b0 2024-03-16T09:39:33.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.593+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3081a0210 con 0x7fd3081022b0 2024-03-16T09:39:33.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.593+0000 7fd2feffd640 1 -- 172.21.15.77:0/2935284106 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7fd30009c180 con 0x7fd3081022b0 2024-03-16T09:39:33.596 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:33.597 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T09:39:33.597 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-16T09:39:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2f40a2800 msgr2=0x7fd2f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2f40a2800 0x7fd2f40a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8006010 tx=0x7fd2f8005be0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3081022b0 msgr2=0x7fd30819ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3081022b0 0x7fd30819ef80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd30000eaa0 tx=0x7fd30000eea0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 shutdown_connections 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2f40a2800 0x7fd2f40a4cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3081022b0 0x7fd30819ef80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3081018f0 0x7fd308197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 --2- 172.21.15.77:0/2935284106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd308100700 0x7fd3081979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 >> 172.21.15.77:0/2935284106 conn(0x7fd3080fbc70 msgr2=0x7fd3080fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 shutdown_connections 2024-03-16T09:39:33.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:33.595+0000 7fd31020b640 1 -- 172.21.15.77:0/2935284106 wait complete. 2024-03-16T09:39:33.990 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:33 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2935284106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:34.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:33 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2935284106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:34.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:33 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2935284106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:34.551 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:39:34 smithi129 systemd[1]: Started Ceph mds.f for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:35.182 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:34 smithi129 ceph-mon[35841]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:35.182 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.182 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.182 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.182 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:35.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:35.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:34 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:34 smithi119 ceph-mon[35632]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:34 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:34 smithi077 ceph-mon[32171]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:35.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:35.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:34 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:35.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:35.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 -- 172.21.15.77:0/2765870549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110101910 msgr2=0x7ff110101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 --2- 172.21.15.77:0/2765870549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110101910 0x7ff110101d90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009900 tx=0x7ff0fc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 -- 172.21.15.77:0/2765870549 shutdown_connections 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 --2- 172.21.15.77:0/2765870549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1101022d0 0x7ff110108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 --2- 172.21.15.77:0/2765870549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110101910 0x7ff110101d90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 --2- 172.21.15.77:0/2765870549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff110100720 0x7ff110100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 -- 172.21.15.77:0/2765870549 >> 172.21.15.77:0/2765870549 conn(0x7ff1100fbc90 msgr2=0x7ff1100fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 -- 172.21.15.77:0/2765870549 shutdown_connections 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.721+0000 7ff116f97640 1 -- 172.21.15.77:0/2765870549 wait complete. 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 Processor -- start 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 -- start start 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110100720 0x7ff110197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff110101910 0x7ff110198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1101022d0 0x7ff11019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:35.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff11010c2b0 con 0x7ff110100720 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff11010c130 con 0x7ff1101022d0 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff116f97640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff11010c430 con 0x7ff110101910 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110100720 0x7ff110197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110100720 0x7ff110197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35436/0 (socket says 172.21.15.77:35436) 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 -- 172.21.15.77:0/80876773 learned_addr learned my addr 172.21.15.77:0/80876773 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff10ffff640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff110101910 0x7ff110198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff115534640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1101022d0 0x7ff11019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 -- 172.21.15.77:0/80876773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff110101910 msgr2=0x7ff110198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff110101910 0x7ff110198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 -- 172.21.15.77:0/80876773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1101022d0 msgr2=0x7ff11019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1101022d0 0x7ff11019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.722+0000 7ff114d33640 1 -- 172.21.15.77:0/80876773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11019f6f0 con 0x7ff110100720 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.723+0000 7ff114d33640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110100720 0x7ff110197ad0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff10400d770 tx=0x7ff10400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:35.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.723+0000 7ff10dffb640 1 -- 172.21.15.77:0/80876773 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104017070 con 0x7ff110100720 2024-03-16T09:39:35.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.723+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11019f9e0 con 0x7ff110100720 2024-03-16T09:39:35.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.723+0000 7ff10dffb640 1 -- 172.21.15.77:0/80876773 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff104009c30 con 0x7ff110100720 2024-03-16T09:39:35.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.723+0000 7ff10dffb640 1 -- 172.21.15.77:0/80876773 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104003d00 con 0x7ff110100720 2024-03-16T09:39:35.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.723+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11019ff80 con 0x7ff110100720 2024-03-16T09:39:35.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.724+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1101a0150 con 0x7ff110100720 2024-03-16T09:39:35.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.725+0000 7ff10dffb640 1 -- 172.21.15.77:0/80876773 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff104003ea0 con 0x7ff110100720 2024-03-16T09:39:35.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.726+0000 7ff10dffb640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff1000a2800 0x7ff1000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:35.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.726+0000 7ff10ffff640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff1000a2800 0x7ff1000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:35.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.726+0000 7ff10dffb640 1 -- 172.21.15.77:0/80876773 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff1040ce2c0 con 0x7ff110100720 2024-03-16T09:39:35.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.726+0000 7ff10ffff640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff1000a2800 0x7ff1000a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009230 tx=0x7ff0fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:35.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.730+0000 7ff10dffb640 1 -- 172.21.15.77:0/80876773 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff10409b800 con 0x7ff110100720 2024-03-16T09:39:35.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.944+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1101a04e0 con 0x7ff110100720 2024-03-16T09:39:35.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.944+0000 7ff10dffb640 1 -- 172.21.15.77:0/80876773 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7ff104009de0 con 0x7ff110100720 2024-03-16T09:39:35.948 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:35.948 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T09:39:35.948 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 7 2024-03-16T09:39:35.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.946+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff1000a2800 msgr2=0x7ff1000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:35.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.946+0000 7ff116f97640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff1000a2800 0x7ff1000a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009230 tx=0x7ff0fc005fd0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.946+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110100720 msgr2=0x7ff110197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.946+0000 7ff116f97640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110100720 0x7ff110197ad0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff10400d770 tx=0x7ff10400dc30 comp rx=0 tx=0).stop 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.946+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 shutdown_connections 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.947+0000 7ff116f97640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff1000a2800 0x7ff1000a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.947+0000 7ff116f97640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1101022d0 0x7ff11019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.947+0000 7ff116f97640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff110101910 0x7ff110198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.947+0000 7ff116f97640 1 --2- 172.21.15.77:0/80876773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff110100720 0x7ff110197ad0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.947+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 >> 172.21.15.77:0/80876773 conn(0x7ff1100fbc90 msgr2=0x7ff1100fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.947+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 shutdown_connections 2024-03-16T09:39:35.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:35.947+0000 7ff116f97640 1 -- 172.21.15.77:0/80876773 wait complete. 2024-03-16T09:39:35.959 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:35 smithi077 ceph-mon[32171]: Deploying daemon mds.c on smithi129 2024-03-16T09:39:35.959 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:35 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6834/2328435031,v1:172.21.15.129:6835/2328435031] up:boot 2024-03-16T09:39:35.959 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:35 smithi077 ceph-mon[32171]: fsmap 6 up:standby 2024-03-16T09:39:35.960 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:35 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T09:39:36.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:35 smithi119 ceph-mon[35632]: Deploying daemon mds.c on smithi129 2024-03-16T09:39:36.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:35 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6834/2328435031,v1:172.21.15.129:6835/2328435031] up:boot 2024-03-16T09:39:36.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:35 smithi119 ceph-mon[35632]: fsmap 6 up:standby 2024-03-16T09:39:36.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:35 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T09:39:36.240 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:35 smithi129 ceph-mon[35841]: Deploying daemon mds.c on smithi129 2024-03-16T09:39:36.240 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:35 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6834/2328435031,v1:172.21.15.129:6835/2328435031] up:boot 2024-03-16T09:39:36.240 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:35 smithi129 ceph-mon[35841]: fsmap 6 up:standby 2024-03-16T09:39:36.240 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:35 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T09:39:36.551 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 systemd[1]: Starting Ceph mds.c for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:36.948 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:36 smithi129 ceph-mon[35841]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:36.948 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:36 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/80876773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:36.948 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 podman[66449]: 2024-03-16 09:39:36.660112222 +0000 UTC m=+0.110608159 container create 6d306348933d820bdf46ae885d187614e22f901265107c449b2758382c847930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T09:39:36.948 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 podman[66449]: 2024-03-16 09:39:36.567608446 +0000 UTC m=+0.018104358 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:36.948 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 podman[66449]: 2024-03-16 09:39:36.854167839 +0000 UTC m=+0.304663739 container init 6d306348933d820bdf46ae885d187614e22f901265107c449b2758382c847930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:39:36.948 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 podman[66449]: 2024-03-16 09:39:36.857005547 +0000 UTC m=+0.307501451 container start 6d306348933d820bdf46ae885d187614e22f901265107c449b2758382c847930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-16T09:39:36.949 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 ceph-mds[66468]: -- 172.21.15.129:0/110212703 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a20a47da40 con 0x55a20a450800 2024-03-16T09:39:36.949 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c[66464]: starting mds.c at 2024-03-16T09:39:36.949 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 bash[66449]: 6d306348933d820bdf46ae885d187614e22f901265107c449b2758382c847930 2024-03-16T09:39:37.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:36 smithi119 ceph-mon[35632]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:37.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:36 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/80876773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:36 smithi077 ceph-mon[32171]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:36 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/80876773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:37.300 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:39:36 smithi129 systemd[1]: Started Ceph mds.c for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:37.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:38.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 -- 172.21.15.77:0/510395131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e80102df0 msgr2=0x7f0e80103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:38.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 --2- 172.21.15.77:0/510395131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e80102df0 0x7f0e80103210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0e78009ca0 tx=0x7f0e7802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 -- 172.21.15.77:0/510395131 shutdown_connections 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 --2- 172.21.15.77:0/510395131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80100220 0x7f0e80109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 --2- 172.21.15.77:0/510395131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e800ff860 0x7f0e800ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 --2- 172.21.15.77:0/510395131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e80102df0 0x7f0e80103210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 -- 172.21.15.77:0/510395131 >> 172.21.15.77:0/510395131 conn(0x7f0e800fb400 msgr2=0x7f0e800fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 -- 172.21.15.77:0/510395131 shutdown_connections 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.073+0000 7f0e87776640 1 -- 172.21.15.77:0/510395131 wait complete. 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 Processor -- start 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 -- start start 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800ff860 0x7f0e80197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e80100220 0x7f0e80197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102df0 0x7f0e8019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8010d320 con 0x7f0e800ff860 2024-03-16T09:39:38.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8010d1a0 con 0x7f0e80102df0 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e87776640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8010d4a0 con 0x7f0e80100220 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800ff860 0x7f0e80197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800ff860 0x7f0e80197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35454/0 (socket says 172.21.15.77:35454) 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 -- 172.21.15.77:0/1921612495 learned_addr learned my addr 172.21.15.77:0/1921612495 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85d13640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102df0 0x7f0e8019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 -- 172.21.15.77:0/1921612495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e80100220 msgr2=0x7f0e80197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e80100220 0x7f0e80197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 -- 172.21.15.77:0/1921612495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102df0 msgr2=0x7f0e8019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102df0 0x7f0e8019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 -- 172.21.15.77:0/1921612495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8019f550 con 0x7f0e800ff860 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.074+0000 7f0e85512640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800ff860 0x7f0e80197930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0e78032380 tx=0x7f0e78032890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.075+0000 7f0e7e7fc640 1 -- 172.21.15.77:0/1921612495 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78041020 con 0x7f0e800ff860 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.075+0000 7f0e7e7fc640 1 -- 172.21.15.77:0/1921612495 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e78003e20 con 0x7f0e800ff860 2024-03-16T09:39:38.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.075+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8019f7e0 con 0x7f0e800ff860 2024-03-16T09:39:38.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.075+0000 7f0e7e7fc640 1 -- 172.21.15.77:0/1921612495 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78039040 con 0x7f0e800ff860 2024-03-16T09:39:38.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.075+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8019fd20 con 0x7f0e800ff860 2024-03-16T09:39:38.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.076+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e80108cb0 con 0x7f0e800ff860 2024-03-16T09:39:38.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.077+0000 7f0e7e7fc640 1 -- 172.21.15.77:0/1921612495 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0e78044020 con 0x7f0e800ff860 2024-03-16T09:39:38.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.077+0000 7f0e7e7fc640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0e680a2880 0x7f0e680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:38.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.078+0000 7f0e84d11640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0e680a2880 0x7f0e680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:38.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.078+0000 7f0e7e7fc640 1 -- 172.21.15.77:0/1921612495 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0e780f0da0 con 0x7f0e800ff860 2024-03-16T09:39:38.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.078+0000 7f0e84d11640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0e680a2880 0x7f0e680a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0e80198e40 tx=0x7f0e74005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:38.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.081+0000 7f0e7e7fc640 1 -- 172.21.15.77:0/1921612495 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e7804b020 con 0x7f0e800ff860 2024-03-16T09:39:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:38.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: Deploying daemon mds.h on smithi119 2024-03-16T09:39:38.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:38.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6836/814033341,v1:172.21.15.129:6837/814033341] up:boot 2024-03-16T09:39:38.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: fsmap 7 up:standby 2024-03-16T09:39:38.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:37 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T09:39:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:38.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:38.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:38.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: Deploying daemon mds.h on smithi119 2024-03-16T09:39:38.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:38.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6836/814033341,v1:172.21.15.129:6837/814033341] up:boot 2024-03-16T09:39:38.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: fsmap 7 up:standby 2024-03-16T09:39:38.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:37 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T09:39:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.293+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e8002d1b0 con 0x7f0e800ff860 2024-03-16T09:39:38.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.293+0000 7f0e7e7fc640 1 -- 172.21.15.77:0/1921612495 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f0e780be2e0 con 0x7f0e800ff860 2024-03-16T09:39:38.296 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:38.296 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T09:39:38.296 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-03-16T09:39:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0e680a2880 msgr2=0x7f0e680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0e680a2880 0x7f0e680a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0e80198e40 tx=0x7f0e74005c70 comp rx=0 tx=0).stop 2024-03-16T09:39:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800ff860 msgr2=0x7f0e80197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800ff860 0x7f0e80197930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0e78032380 tx=0x7f0e78032890 comp rx=0 tx=0).stop 2024-03-16T09:39:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 shutdown_connections 2024-03-16T09:39:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0e680a2880 0x7f0e680a4d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102df0 0x7f0e8019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e80100220 0x7f0e80197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 --2- 172.21.15.77:0/1921612495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800ff860 0x7f0e80197930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 >> 172.21.15.77:0/1921612495 conn(0x7f0e800fb400 msgr2=0x7f0e80105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 shutdown_connections 2024-03-16T09:39:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:38.295+0000 7f0e87776640 1 -- 172.21.15.77:0/1921612495 wait complete. 2024-03-16T09:39:38.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: Deploying daemon mds.h on smithi119 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6836/814033341,v1:172.21.15.129:6837/814033341] up:boot 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: fsmap 7 up:standby 2024-03-16T09:39:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:37 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T09:39:39.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:39 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1921612495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:39.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:39 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1921612495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:39.411 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 systemd[1]: Starting Ceph mds.h for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:39 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1921612495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:39.674 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 podman[65705]: 2024-03-16 09:39:39.311433539 +0000 UTC m=+0.017720790 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:39.674 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 podman[65705]: 2024-03-16 09:39:39.41946872 +0000 UTC m=+0.125755967 container create 61a7a126692ec6998ce84b93e9165f38642f493611e365948475fbaa6a0dbfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-16T09:39:39.675 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 podman[65705]: 2024-03-16 09:39:39.672226037 +0000 UTC m=+0.378513293 container init 61a7a126692ec6998ce84b93e9165f38642f493611e365948475fbaa6a0dbfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-16T09:39:39.940 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 podman[65705]: 2024-03-16 09:39:39.674912984 +0000 UTC m=+0.381200231 container start 61a7a126692ec6998ce84b93e9165f38642f493611e365948475fbaa6a0dbfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T09:39:39.940 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 ceph-mds[65724]: -- 172.21.15.119:0/3169198336 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d76e80fa40 con 0x55d76e7e2c00 2024-03-16T09:39:39.940 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h[65720]: starting mds.h at 2024-03-16T09:39:39.940 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 bash[65705]: 61a7a126692ec6998ce84b93e9165f38642f493611e365948475fbaa6a0dbfe6 2024-03-16T09:39:39.940 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:39:39 smithi119 systemd[1]: Started Ceph mds.h for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:40.310 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/4123045676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c81022c0 msgr2=0x7fe3c8108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/4123045676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c81022c0 0x7fe3c8108b50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009ca0 tx=0x7fe3c002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/4123045676 shutdown_connections 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/4123045676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c81022c0 0x7fe3c8108b50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/4123045676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3c8101900 0x7fe3c8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/4123045676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3c8100710 0x7fe3c8100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/4123045676 >> 172.21.15.77:0/4123045676 conn(0x7fe3c80fbcc0 msgr2=0x7fe3c80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/4123045676 shutdown_connections 2024-03-16T09:39:40.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.420+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/4123045676 wait complete. 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 Processor -- start 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 -- start start 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3c8100710 0x7fe3c806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c8101900 0x7fe3c819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3c81022c0 0x7fe3c819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c810c240 con 0x7fe3c8101900 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c810c0c0 con 0x7fe3c81022c0 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3ce0a1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c810c3c0 con 0x7fe3c8100710 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c8101900 0x7fe3c819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c8101900 0x7fe3c819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35468/0 (socket says 172.21.15.77:35468) 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 -- 172.21.15.77:0/224924744 learned_addr learned my addr 172.21.15.77:0/224924744 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c77fe640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3c8100710 0x7fe3c806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c7fff640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3c81022c0 0x7fe3c819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:40.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 -- 172.21.15.77:0/224924744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3c8100710 msgr2=0x7fe3c806a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3c8100710 0x7fe3c806a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 -- 172.21.15.77:0/224924744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3c81022c0 msgr2=0x7fe3c819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3c81022c0 0x7fe3c819ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.421+0000 7fe3c6ffd640 1 -- 172.21.15.77:0/224924744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c819f640 con 0x7fe3c8101900 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.422+0000 7fe3c6ffd640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c8101900 0x7fe3c819c740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00d950 tx=0x7fe3bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.422+0000 7fe3c4ff9640 1 -- 172.21.15.77:0/224924744 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc017070 con 0x7fe3c8101900 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.422+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c819f930 con 0x7fe3c8101900 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.422+0000 7fe3c4ff9640 1 -- 172.21.15.77:0/224924744 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3bc009c30 con 0x7fe3c8101900 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.422+0000 7fe3c4ff9640 1 -- 172.21.15.77:0/224924744 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00f620 con 0x7fe3c8101900 2024-03-16T09:39:40.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.422+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c819fe80 con 0x7fe3c8101900 2024-03-16T09:39:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.423+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c8107a80 con 0x7fe3c8101900 2024-03-16T09:39:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.424+0000 7fe3c4ff9640 1 -- 172.21.15.77:0/224924744 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe3bc0109a0 con 0x7fe3c8101900 2024-03-16T09:39:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.424+0000 7fe3c4ff9640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe3b00a2800 0x7fe3b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.424+0000 7fe3c77fe640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe3b00a2800 0x7fe3b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.424+0000 7fe3c4ff9640 1 -- 172.21.15.77:0/224924744 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe3bc0ce210 con 0x7fe3c8101900 2024-03-16T09:39:40.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.425+0000 7fe3c77fe640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe3b00a2800 0x7fe3b00a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80058c0 tx=0x7fe3b800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:40.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.429+0000 7fe3c4ff9640 1 -- 172.21.15.77:0/224924744 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3bc009de0 con 0x7fe3c8101900 2024-03-16T09:39:40.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.641+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3c802d1b0 con 0x7fe3c8101900 2024-03-16T09:39:40.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.641+0000 7fe3c4ff9640 1 -- 172.21.15.77:0/224924744 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7fe3bc09b750 con 0x7fe3c8101900 2024-03-16T09:39:40.644 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:40.644 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T09:39:40.645 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-03-16T09:39:40.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe3b00a2800 msgr2=0x7fe3b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:40.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe3b00a2800 0x7fe3b00a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80058c0 tx=0x7fe3b800a2d0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c8101900 msgr2=0x7fe3c819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:40.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c8101900 0x7fe3c819c740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00d950 tx=0x7fe3bc00de10 comp rx=0 tx=0).stop 2024-03-16T09:39:40.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 shutdown_connections 2024-03-16T09:39:40.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe3b00a2800 0x7fe3b00a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3c81022c0 0x7fe3c819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c8101900 0x7fe3c819c740 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 --2- 172.21.15.77:0/224924744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3c8100710 0x7fe3c806a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:40.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.643+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 >> 172.21.15.77:0/224924744 conn(0x7fe3c80fbcc0 msgr2=0x7fe3c8104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:40.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.644+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 shutdown_connections 2024-03-16T09:39:40.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:40.644+0000 7fe3ce0a1640 1 -- 172.21.15.77:0/224924744 wait complete. 2024-03-16T09:39:41.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:41.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: Deploying daemon mds.i on smithi129 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] up:boot 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: fsmap 8 up:standby 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T09:39:41.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:40 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/224924744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:41.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: Deploying daemon mds.i on smithi129 2024-03-16T09:39:41.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] up:boot 2024-03-16T09:39:41.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: fsmap 8 up:standby 2024-03-16T09:39:41.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T09:39:41.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:40 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/224924744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: Deploying daemon mds.i on smithi129 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] up:boot 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: fsmap 8 up:standby 2024-03-16T09:39:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T09:39:41.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:40 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/224924744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:42.051 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:41 smithi129 systemd[1]: Starting Ceph mds.i for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:42.309 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 podman[66755]: 2024-03-16 09:39:42.08780957 +0000 UTC m=+0.110747967 container create 2569ce7aa2f2e4d6948b50ce59db3e312eb11bd7ca8b6dcba4fd791dd719f1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:39:42.309 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 podman[66755]: 2024-03-16 09:39:41.994927534 +0000 UTC m=+0.017865929 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:42.309 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 podman[66755]: 2024-03-16 09:39:42.30679743 +0000 UTC m=+0.329735823 container init 2569ce7aa2f2e4d6948b50ce59db3e312eb11bd7ca8b6dcba4fd791dd719f1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-16T09:39:42.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:42.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 -- 172.21.15.77:0/582491353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 msgr2=0x7fcb9c101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 --2- 172.21.15.77:0/582491353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c101d70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009900 tx=0x7fcb9002bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 -- 172.21.15.77:0/582491353 shutdown_connections 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 --2- 172.21.15.77:0/582491353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb9c1022b0 0x7fcb9c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 --2- 172.21.15.77:0/582491353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c101d70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 --2- 172.21.15.77:0/582491353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb9c100700 0x7fcb9c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 -- 172.21.15.77:0/582491353 >> 172.21.15.77:0/582491353 conn(0x7fcb9c0fbc70 msgr2=0x7fcb9c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 -- 172.21.15.77:0/582491353 shutdown_connections 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 -- 172.21.15.77:0/582491353 wait complete. 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.759+0000 7fcba22c0640 1 Processor -- start 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba22c0640 1 -- start start 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba22c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb9c100700 0x7fcb9c069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba22c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba22c0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb9c1022b0 0x7fcb9c19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:42.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba22c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c10c2a0 con 0x7fcb9c1018f0 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba22c0640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c10c120 con 0x7fcb9c1022b0 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba22c0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c10c420 con 0x7fcb9c100700 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c19c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35492/0 (socket says 172.21.15.77:35492) 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 -- 172.21.15.77:0/1891742672 learned_addr learned my addr 172.21.15.77:0/1891742672 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcba085d640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb9c1022b0 0x7fcb9c19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9bfff640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb9c100700 0x7fcb9c069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 -- 172.21.15.77:0/1891742672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb9c100700 msgr2=0x7fcb9c069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb9c100700 0x7fcb9c069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 -- 172.21.15.77:0/1891742672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb9c1022b0 msgr2=0x7fcb9c19ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb9c1022b0 0x7fcb9c19ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 -- 172.21.15.77:0/1891742672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9c19f740 con 0x7fcb9c1018f0 2024-03-16T09:39:42.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.760+0000 7fcb9b7fe640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c19c810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcb90006010 tx=0x7fcb900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:42.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.761+0000 7fcb997fa640 1 -- 172.21.15.77:0/1891742672 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90043020 con 0x7fcb9c1018f0 2024-03-16T09:39:42.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.761+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9c19f9d0 con 0x7fcb9c1018f0 2024-03-16T09:39:42.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.761+0000 7fcb997fa640 1 -- 172.21.15.77:0/1891742672 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb90004aa0 con 0x7fcb9c1018f0 2024-03-16T09:39:42.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.761+0000 7fcb997fa640 1 -- 172.21.15.77:0/1891742672 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90003c60 con 0x7fcb9c1018f0 2024-03-16T09:39:42.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.761+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9c19ff70 con 0x7fcb9c1018f0 2024-03-16T09:39:42.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.762+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb9c107ae0 con 0x7fcb9c1018f0 2024-03-16T09:39:42.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.763+0000 7fcb997fa640 1 -- 172.21.15.77:0/1891742672 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcb90046020 con 0x7fcb9c1018f0 2024-03-16T09:39:42.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.763+0000 7fcb997fa640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb840a2800 0x7fcb840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:42.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.763+0000 7fcb9bfff640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb840a2800 0x7fcb840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:42.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.764+0000 7fcb997fa640 1 -- 172.21.15.77:0/1891742672 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcb900f09c0 con 0x7fcb9c1018f0 2024-03-16T09:39:42.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.764+0000 7fcb9bfff640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb840a2800 0x7fcb840a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c101750 tx=0x7fcb8c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:42.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:42.772+0000 7fcb997fa640 1 -- 172.21.15.77:0/1891742672 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb900f2e20 con 0x7fcb9c1018f0 2024-03-16T09:39:42.801 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 podman[66755]: 2024-03-16 09:39:42.30960078 +0000 UTC m=+0.332539174 container start 2569ce7aa2f2e4d6948b50ce59db3e312eb11bd7ca8b6dcba4fd791dd719f1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:39:42.801 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mds[66775]: -- 172.21.15.129:0/2116672667 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b8c1d7a40 con 0x564b8c1ab000 2024-03-16T09:39:42.801 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i[66771]: starting mds.i at 2024-03-16T09:39:42.801 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 bash[66755]: 2569ce7aa2f2e4d6948b50ce59db3e312eb11bd7ca8b6dcba4fd791dd719f1e7 2024-03-16T09:39:42.801 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:39:42 smithi129 systemd[1]: Started Ceph mds.i for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:43.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.005+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb9c02d400 con 0x7fcb9c1018f0 2024-03-16T09:39:43.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.005+0000 7fcb997fa640 1 -- 172.21.15.77:0/1891742672 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7fcb9004d020 con 0x7fcb9c1018f0 2024-03-16T09:39:43.008 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:43.008 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T09:39:43.009 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 10 2024-03-16T09:39:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:42 smithi077 ceph-mon[32171]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:42 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:42 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:42 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:42 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:42 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:42 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:43.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb840a2800 msgr2=0x7fcb840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:43.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb840a2800 0x7fcb840a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c101750 tx=0x7fcb8c008040 comp rx=0 tx=0).stop 2024-03-16T09:39:43.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 msgr2=0x7fcb9c19c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:43.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c19c810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcb90006010 tx=0x7fcb900042f0 comp rx=0 tx=0).stop 2024-03-16T09:39:43.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 shutdown_connections 2024-03-16T09:39:43.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb840a2800 0x7fcb840a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:43.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb9c1022b0 0x7fcb9c19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb9c1018f0 0x7fcb9c19c810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 --2- 172.21.15.77:0/1891742672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb9c100700 0x7fcb9c069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 >> 172.21.15.77:0/1891742672 conn(0x7fcb9c0fbc70 msgr2=0x7fcb9c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 shutdown_connections 2024-03-16T09:39:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:43.008+0000 7fcba22c0640 1 -- 172.21.15.77:0/1891742672 wait complete. 2024-03-16T09:39:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:42 smithi119 ceph-mon[35632]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:42 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:42 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:42 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:42 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:42 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:42 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mon[35841]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:43.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:42 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:43 smithi129 ceph-mon[35841]: Deploying daemon mds.g on smithi077 2024-03-16T09:39:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:43 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] up:boot 2024-03-16T09:39:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:43 smithi129 ceph-mon[35841]: fsmap 9 up:standby 2024-03-16T09:39:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:43 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T09:39:44.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:43 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1891742672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:44.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:43 smithi077 ceph-mon[32171]: Deploying daemon mds.g on smithi077 2024-03-16T09:39:44.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:43 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] up:boot 2024-03-16T09:39:44.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:43 smithi077 ceph-mon[32171]: fsmap 9 up:standby 2024-03-16T09:39:44.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:43 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T09:39:44.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:43 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1891742672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:44.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:43 smithi119 ceph-mon[35632]: Deploying daemon mds.g on smithi077 2024-03-16T09:39:44.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:43 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] up:boot 2024-03-16T09:39:44.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:43 smithi119 ceph-mon[35632]: fsmap 9 up:standby 2024-03-16T09:39:44.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:43 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T09:39:44.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:43 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1891742672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:44.759 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:44 smithi077 systemd[1]: Starting Ceph mds.g for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:45.026 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:45.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:44 smithi129 ceph-mon[35841]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:45.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 -- 172.21.15.77:0/2128599697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3681022b0 msgr2=0x7fb368108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:45.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 --2- 172.21.15.77:0/2128599697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3681022b0 0x7fb368108b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb360009ca0 tx=0x7fb36002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 -- 172.21.15.77:0/2128599697 shutdown_connections 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 --2- 172.21.15.77:0/2128599697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3681022b0 0x7fb368108b40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 --2- 172.21.15.77:0/2128599697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3681018f0 0x7fb368101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 --2- 172.21.15.77:0/2128599697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb368100700 0x7fb368100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 -- 172.21.15.77:0/2128599697 >> 172.21.15.77:0/2128599697 conn(0x7fb3680fbc70 msgr2=0x7fb3680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 -- 172.21.15.77:0/2128599697 shutdown_connections 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.134+0000 7fb36ce68640 1 -- 172.21.15.77:0/2128599697 wait complete. 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 Processor -- start 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 -- start start 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb368100700 0x7fb368199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3681018f0 0x7fb36819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3681022b0 0x7fb3681a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:45.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810c2a0 con 0x7fb368100700 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810c120 con 0x7fb3681018f0 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36ce68640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810c420 con 0x7fb3681022b0 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb368100700 0x7fb368199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb368100700 0x7fb368199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35498/0 (socket says 172.21.15.77:35498) 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 -- 172.21.15.77:0/638791763 learned_addr learned my addr 172.21.15.77:0/638791763 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb366d9d640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3681022b0 0x7fb3681a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb365d9b640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3681018f0 0x7fb36819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 -- 172.21.15.77:0/638791763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3681022b0 msgr2=0x7fb3681a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3681022b0 0x7fb3681a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 -- 172.21.15.77:0/638791763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3681018f0 msgr2=0x7fb36819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3681018f0 0x7fb36819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 -- 172.21.15.77:0/638791763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3681a1810 con 0x7fb368100700 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb365d9b640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3681018f0 0x7fb36819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:39:45.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.135+0000 7fb36659c640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb368100700 0x7fb368199bf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb35800b450 tx=0x7fb35800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:45.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.136+0000 7fb3577fe640 1 -- 172.21.15.77:0/638791763 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358015020 con 0x7fb368100700 2024-03-16T09:39:45.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.136+0000 7fb3577fe640 1 -- 172.21.15.77:0/638791763 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb358011d40 con 0x7fb368100700 2024-03-16T09:39:45.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.136+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3681a1b00 con 0x7fb368100700 2024-03-16T09:39:45.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.136+0000 7fb3577fe640 1 -- 172.21.15.77:0/638791763 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35800f340 con 0x7fb368100700 2024-03-16T09:39:45.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.136+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3681a2050 con 0x7fb368100700 2024-03-16T09:39:45.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.137+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3681a2280 con 0x7fb368100700 2024-03-16T09:39:45.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.138+0000 7fb3577fe640 1 -- 172.21.15.77:0/638791763 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb358018020 con 0x7fb368100700 2024-03-16T09:39:45.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.138+0000 7fb3577fe640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3500a2800 0x7fb3500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:45.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.138+0000 7fb365d9b640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3500a2800 0x7fb3500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:45.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.138+0000 7fb3577fe640 1 -- 172.21.15.77:0/638791763 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb3580ce5d0 con 0x7fb368100700 2024-03-16T09:39:45.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.139+0000 7fb365d9b640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3500a2800 0x7fb3500a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb36819b100 tx=0x7fb35c005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:45.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.142+0000 7fb3577fe640 1 -- 172.21.15.77:0/638791763 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb35801f020 con 0x7fb368100700 2024-03-16T09:39:45.165 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:44 smithi077 podman[73891]: 2024-03-16 09:39:44.801274412 +0000 UTC m=+0.018625785 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:45.165 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:44 smithi077 podman[73891]: 2024-03-16 09:39:44.909653708 +0000 UTC m=+0.127005074 container create b8e1b98ba418613b54111c4d8e83693c4053715e5d811ec4886dc21d59f41390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T09:39:45.165 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:45 smithi077 podman[73891]: 2024-03-16 09:39:45.162806479 +0000 UTC m=+0.380157851 container init b8e1b98ba418613b54111c4d8e83693c4053715e5d811ec4886dc21d59f41390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T09:39:45.165 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:44 smithi077 ceph-mon[32171]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:44 smithi119 ceph-mon[35632]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:45.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.384+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3681a2450 con 0x7fb368100700 2024-03-16T09:39:45.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.385+0000 7fb3577fe640 1 -- 172.21.15.77:0/638791763 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7fb35800d040 con 0x7fb368100700 2024-03-16T09:39:45.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:45.388 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T09:39:45.388 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 10 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3500a2800 msgr2=0x7fb3500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3500a2800 0x7fb3500a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb36819b100 tx=0x7fb35c005d90 comp rx=0 tx=0).stop 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb368100700 msgr2=0x7fb368199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb368100700 0x7fb368199bf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb35800b450 tx=0x7fb35800b910 comp rx=0 tx=0).stop 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 shutdown_connections 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3500a2800 0x7fb3500a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3681022b0 0x7fb3681a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3681018f0 0x7fb36819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 --2- 172.21.15.77:0/638791763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb368100700 0x7fb368199bf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 >> 172.21.15.77:0/638791763 conn(0x7fb3680fbc70 msgr2=0x7fb3680fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 shutdown_connections 2024-03-16T09:39:45.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:45.387+0000 7fb36ce68640 1 -- 172.21.15.77:0/638791763 wait complete. 2024-03-16T09:39:45.490 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:45 smithi077 podman[73891]: 2024-03-16 09:39:45.166767539 +0000 UTC m=+0.384118906 container start b8e1b98ba418613b54111c4d8e83693c4053715e5d811ec4886dc21d59f41390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-16T09:39:45.490 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:45 smithi077 ceph-mds[73950]: -- 172.21.15.77:0/92562247 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556275597a40 con 0x55627556ac00 2024-03-16T09:39:45.490 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g[73913]: starting mds.g at 2024-03-16T09:39:45.490 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:45 smithi077 bash[73891]: b8e1b98ba418613b54111c4d8e83693c4053715e5d811ec4886dc21d59f41390 2024-03-16T09:39:45.490 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:39:45 smithi077 systemd[1]: Started Ceph mds.g for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: Deploying daemon mds.a on smithi077 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/638791763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:46.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:46.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.77:6838/1432550583,v1:172.21.15.77:6839/1432550583] up:boot 2024-03-16T09:39:46.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: fsmap 10 up:standby 2024-03-16T09:39:46.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:46 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T09:39:46.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: Deploying daemon mds.a on smithi077 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/638791763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.77:6838/1432550583,v1:172.21.15.77:6839/1432550583] up:boot 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: fsmap 10 up:standby 2024-03-16T09:39:46.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:46 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: Deploying daemon mds.a on smithi077 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/638791763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:46.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:46.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.77:6838/1432550583,v1:172.21.15.77:6839/1432550583] up:boot 2024-03-16T09:39:46.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: fsmap 10 up:standby 2024-03-16T09:39:46.802 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:46 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T09:39:47.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:47.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 -- 172.21.15.77:0/3507253852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc41022d0 msgr2=0x7f6fc4108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:47.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 --2- 172.21.15.77:0/3507253852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc41022d0 0x7f6fc4108b60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009ca0 tx=0x7f6fbc02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 -- 172.21.15.77:0/3507253852 shutdown_connections 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 --2- 172.21.15.77:0/3507253852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc41022d0 0x7f6fc4108b60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 --2- 172.21.15.77:0/3507253852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fc4101910 0x7f6fc4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 --2- 172.21.15.77:0/3507253852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc4100720 0x7f6fc4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 -- 172.21.15.77:0/3507253852 >> 172.21.15.77:0/3507253852 conn(0x7f6fc40fbc90 msgr2=0x7f6fc40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 -- 172.21.15.77:0/3507253852 shutdown_connections 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 -- 172.21.15.77:0/3507253852 wait complete. 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.515+0000 7f6fc926e640 1 Processor -- start 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc926e640 1 -- start start 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc926e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc4100720 0x7f6fc406a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc926e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc4101910 0x7f6fc419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc926e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fc41022d0 0x7f6fc419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc926e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc410c250 con 0x7f6fc4101910 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc926e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc410c0d0 con 0x7f6fc41022d0 2024-03-16T09:39:47.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc926e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc410c3d0 con 0x7f6fc4100720 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc4101910 0x7f6fc419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc4101910 0x7f6fc419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52786/0 (socket says 172.21.15.77:52786) 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 -- 172.21.15.77:0/1654735422 learned_addr learned my addr 172.21.15.77:0/1654735422 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc37fe640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fc41022d0 0x7f6fc419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc2ffd640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc4100720 0x7f6fc406a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 -- 172.21.15.77:0/1654735422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc4100720 msgr2=0x7f6fc406a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc4100720 0x7f6fc406a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 -- 172.21.15.77:0/1654735422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fc41022d0 msgr2=0x7f6fc419ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fc41022d0 0x7f6fc419ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 -- 172.21.15.77:0/1654735422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc419f6a0 con 0x7f6fc4101910 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fc27fc640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc4101910 0x7f6fc419c7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400d950 tx=0x7f6fb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.516+0000 7f6fabfff640 1 -- 172.21.15.77:0/1654735422 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4017030 con 0x7f6fc4101910 2024-03-16T09:39:47.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.517+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc419f990 con 0x7f6fc4101910 2024-03-16T09:39:47.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.517+0000 7f6fabfff640 1 -- 172.21.15.77:0/1654735422 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fb4004020 con 0x7f6fc4101910 2024-03-16T09:39:47.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.517+0000 7f6fabfff640 1 -- 172.21.15.77:0/1654735422 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb40047a0 con 0x7f6fc4101910 2024-03-16T09:39:47.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.517+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc419ff30 con 0x7f6fc4101910 2024-03-16T09:39:47.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.518+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc4107b60 con 0x7f6fc4101910 2024-03-16T09:39:47.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.519+0000 7f6fabfff640 1 -- 172.21.15.77:0/1654735422 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6fb40041d0 con 0x7f6fc4101910 2024-03-16T09:39:47.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.519+0000 7f6fabfff640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6fac0a2800 0x7f6fac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:47.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.520+0000 7f6fabfff640 1 -- 172.21.15.77:0/1654735422 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6fb40ce100 con 0x7f6fc4101910 2024-03-16T09:39:47.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.520+0000 7f6fc2ffd640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6fac0a2800 0x7f6fac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:47.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.520+0000 7f6fc2ffd640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6fac0a2800 0x7f6fac0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8005fd0 tx=0x7f6fb8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:47.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.524+0000 7f6fabfff640 1 -- 172.21.15.77:0/1654735422 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6fb409b640 con 0x7f6fc4101910 2024-03-16T09:39:47.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.756+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fc402d1b0 con 0x7f6fc4101910 2024-03-16T09:39:47.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.757+0000 7f6fabfff640 1 -- 172.21.15.77:0/1654735422 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f6fb409af40 con 0x7f6fc4101910 2024-03-16T09:39:47.760 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:47.760 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T09:39:47.760 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 11 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6fac0a2800 msgr2=0x7f6fac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6fac0a2800 0x7f6fac0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8005fd0 tx=0x7f6fb8005850 comp rx=0 tx=0).stop 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc4101910 msgr2=0x7f6fc419c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc4101910 0x7f6fc419c7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400d950 tx=0x7f6fb400de10 comp rx=0 tx=0).stop 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 shutdown_connections 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6fac0a2800 0x7f6fac0a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fc41022d0 0x7f6fc419ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc4101910 0x7f6fc419c7a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 --2- 172.21.15.77:0/1654735422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fc4100720 0x7f6fc406a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:47.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 >> 172.21.15.77:0/1654735422 conn(0x7f6fc40fbc90 msgr2=0x7f6fc4104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:47.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 shutdown_connections 2024-03-16T09:39:47.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:47.759+0000 7f6fc926e640 1 -- 172.21.15.77:0/1654735422 wait complete. 2024-03-16T09:39:47.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:47 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1654735422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:48.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:47 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1654735422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:48.259 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 09:39:47 smithi077 systemd[1]: Started Ceph mds.a for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:48.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:47 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1654735422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:48.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:48 smithi119 ceph-mon[35632]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:48.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:48.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:48.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:48.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:48.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:48.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:48 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:48 smithi077 ceph-mon[32171]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:48 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:48 smithi129 ceph-mon[35841]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T09:39:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T09:39:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:48 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:49.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.886+0000 7ff030787640 1 -- 172.21.15.77:0/2440295985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0281022e0 msgr2=0x7ff0281046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.886+0000 7ff030787640 1 --2- 172.21.15.77:0/2440295985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0281022e0 0x7ff0281046d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff024009900 tx=0x7ff02402bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.886+0000 7ff030787640 1 -- 172.21.15.77:0/2440295985 shutdown_connections 2024-03-16T09:39:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.886+0000 7ff030787640 1 --2- 172.21.15.77:0/2440295985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff028107540 0x7ff028109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.886+0000 7ff030787640 1 --2- 172.21.15.77:0/2440295985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff028104c10 0x7ff028107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.886+0000 7ff030787640 1 --2- 172.21.15.77:0/2440295985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0281022e0 0x7ff0281046d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.886+0000 7ff030787640 1 -- 172.21.15.77:0/2440295985 >> 172.21.15.77:0/2440295985 conn(0x7ff0280fbc70 msgr2=0x7ff0280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 -- 172.21.15.77:0/2440295985 shutdown_connections 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 -- 172.21.15.77:0/2440295985 wait complete. 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 Processor -- start 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 -- start start 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0281022e0 0x7ff028197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff028104c10 0x7ff028197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff028107540 0x7ff02819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff02810d320 con 0x7ff028104c10 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff02810d1a0 con 0x7ff0281022e0 2024-03-16T09:39:49.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff030787640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff02810d4a0 con 0x7ff028107540 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff02dd22640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff028104c10 0x7ff028197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff02dd22640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff028104c10 0x7ff028197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52816/0 (socket says 172.21.15.77:52816) 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff02dd22640 1 -- 172.21.15.77:0/899143816 learned_addr learned my addr 172.21.15.77:0/899143816 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.887+0000 7ff02e523640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0281022e0 0x7ff028197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff02ed24640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff028107540 0x7ff02819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff02dd22640 1 -- 172.21.15.77:0/899143816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff028107540 msgr2=0x7ff02819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff02dd22640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff028107540 0x7ff02819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff02dd22640 1 -- 172.21.15.77:0/899143816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0281022e0 msgr2=0x7ff028197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff02dd22640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0281022e0 0x7ff028197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff02dd22640 1 -- 172.21.15.77:0/899143816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff02819f620 con 0x7ff028104c10 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff02dd22640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff028104c10 0x7ff028197f40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff01800e9c0 tx=0x7ff01800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:49.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff01f7fe640 1 -- 172.21.15.77:0/899143816 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01800cd80 con 0x7ff028104c10 2024-03-16T09:39:49.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff01f7fe640 1 -- 172.21.15.77:0/899143816 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0180054d0 con 0x7ff028104c10 2024-03-16T09:39:49.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff02819f910 con 0x7ff028104c10 2024-03-16T09:39:49.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff01f7fe640 1 -- 172.21.15.77:0/899143816 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018010430 con 0x7ff028104c10 2024-03-16T09:39:49.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.888+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff02819fe60 con 0x7ff028104c10 2024-03-16T09:39:49.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.889+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0280fd760 con 0x7ff028104c10 2024-03-16T09:39:49.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.890+0000 7ff01f7fe640 1 -- 172.21.15.77:0/899143816 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff018004fb0 con 0x7ff028104c10 2024-03-16T09:39:49.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.891+0000 7ff01f7fe640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff0140a2800 0x7ff0140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:49.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.891+0000 7ff02e523640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff0140a2800 0x7ff0140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:49.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.891+0000 7ff01f7fe640 1 -- 172.21.15.77:0/899143816 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff0180cec90 con 0x7ff028104c10 2024-03-16T09:39:49.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.891+0000 7ff02e523640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff0140a2800 0x7ff0140a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff024009230 tx=0x7ff024005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:49.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:49.894+0000 7ff01f7fe640 1 -- 172.21.15.77:0/899143816 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff018053740 con 0x7ff028104c10 2024-03-16T09:39:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:49 smithi119 ceph-mon[35632]: Deploying daemon mds.b on smithi119 2024-03-16T09:39:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:49 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.77:6840/3585324919,v1:172.21.15.77:6841/3585324919] up:boot 2024-03-16T09:39:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:49 smithi119 ceph-mon[35632]: fsmap 11 up:standby 2024-03-16T09:39:49.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:49 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T09:39:50.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.104+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff02802d1b0 con 0x7ff028104c10 2024-03-16T09:39:50.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.105+0000 7ff01f7fe640 1 -- 172.21.15.77:0/899143816 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7ff02802d1b0 con 0x7ff028104c10 2024-03-16T09:39:50.108 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:50.108 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T09:39:50.108 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 12 2024-03-16T09:39:50.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff0140a2800 msgr2=0x7ff0140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:50.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff0140a2800 0x7ff0140a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff024009230 tx=0x7ff024005fd0 comp rx=0 tx=0).stop 2024-03-16T09:39:50.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff028104c10 msgr2=0x7ff028197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:50.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff028104c10 0x7ff028197f40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff01800e9c0 tx=0x7ff01800ee80 comp rx=0 tx=0).stop 2024-03-16T09:39:50.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 shutdown_connections 2024-03-16T09:39:50.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff0140a2800 0x7ff0140a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff028107540 0x7ff02819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff028104c10 0x7ff028197f40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 --2- 172.21.15.77:0/899143816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0281022e0 0x7ff028197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 >> 172.21.15.77:0/899143816 conn(0x7ff0280fbc70 msgr2=0x7ff028102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 shutdown_connections 2024-03-16T09:39:50.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:50.107+0000 7ff030787640 1 -- 172.21.15.77:0/899143816 wait complete. 2024-03-16T09:39:50.119 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:49 smithi077 ceph-mon[32171]: Deploying daemon mds.b on smithi119 2024-03-16T09:39:50.120 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:49 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.77:6840/3585324919,v1:172.21.15.77:6841/3585324919] up:boot 2024-03-16T09:39:50.120 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:49 smithi077 ceph-mon[32171]: fsmap 11 up:standby 2024-03-16T09:39:50.120 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:49 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T09:39:50.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:49 smithi129 ceph-mon[35841]: Deploying daemon mds.b on smithi119 2024-03-16T09:39:50.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:49 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.77:6840/3585324919,v1:172.21.15.77:6841/3585324919] up:boot 2024-03-16T09:39:50.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:49 smithi129 ceph-mon[35841]: fsmap 11 up:standby 2024-03-16T09:39:50.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:49 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T09:39:50.350 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 systemd[1]: Starting Ceph mds.b for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:39:50.604 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 podman[66028]: 2024-03-16 09:39:50.348066697 +0000 UTC m=+0.103455228 container create ccf45c290e5ac6db618ab9c8a06e7ee690dfe21615382df7b324a37d7bd9eb81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T09:39:50.604 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 podman[66028]: 2024-03-16 09:39:50.261979188 +0000 UTC m=+0.017367723 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:39:50.604 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 podman[66028]: 2024-03-16 09:39:50.576201362 +0000 UTC m=+0.331589894 container init ccf45c290e5ac6db618ab9c8a06e7ee690dfe21615382df7b324a37d7bd9eb81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T09:39:50.604 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 podman[66028]: 2024-03-16 09:39:50.579219399 +0000 UTC m=+0.334607929 container start ccf45c290e5ac6db618ab9c8a06e7ee690dfe21615382df7b324a37d7bd9eb81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:39:50.604 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mds[66047]: -- 172.21.15.119:0/2576545063 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b1614da40 con 0x560b16121000 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: starting mds.b at 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 bash[66028]: ccf45c290e5ac6db618ab9c8a06e7ee690dfe21615382df7b324a37d7bd9eb81 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:39:50 smithi119 systemd[1]: Started Ceph mds.b for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mon[35632]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/899143816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:50.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:50.941 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:50 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:50 smithi077 ceph-mon[32171]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:50 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/899143816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:50 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:50 smithi129 ceph-mon[35841]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:50 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/899143816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:50 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:51.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:51 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] up:boot 2024-03-16T09:39:51.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:51 smithi119 ceph-mon[35632]: fsmap 12 up:standby 2024-03-16T09:39:51.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:51 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T09:39:51.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:51 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:51 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] up:boot 2024-03-16T09:39:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:51 smithi077 ceph-mon[32171]: fsmap 12 up:standby 2024-03-16T09:39:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:51 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T09:39:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:51 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:52.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:51 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] up:boot 2024-03-16T09:39:52.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:51 smithi129 ceph-mon[35841]: fsmap 12 up:standby 2024-03-16T09:39:52.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:51 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T09:39:52.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:51 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:52.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:39:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 -- 172.21.15.77:0/2623765227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb01033c0 msgr2=0x7f1bb0109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/2623765227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb01033c0 0x7f1bb0109c50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009ca0 tx=0x7f1ba802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 -- 172.21.15.77:0/2623765227 shutdown_connections 2024-03-16T09:39:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/2623765227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb01033c0 0x7f1bb0109c50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/2623765227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb0102a00 0x7f1bb0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/2623765227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bb0101810 0x7f1bb0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 -- 172.21.15.77:0/2623765227 >> 172.21.15.77:0/2623765227 conn(0x7f1bb00fcd80 msgr2=0x7f1bb00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 -- 172.21.15.77:0/2623765227 shutdown_connections 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.238+0000 7f1bb71f4640 1 -- 172.21.15.77:0/2623765227 wait complete. 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 Processor -- start 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 -- start start 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb0101810 0x7f1bb01979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bb0102a00 0x7f1bb0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb01033c0 0x7f1bb019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb010d320 con 0x7f1bb0101810 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb010d1a0 con 0x7f1bb01033c0 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb71f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb010d4a0 con 0x7f1bb0102a00 2024-03-16T09:39:52.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb0101810 0x7f1bb01979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb0101810 0x7f1bb01979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52836/0 (socket says 172.21.15.77:52836) 2024-03-16T09:39:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 -- 172.21.15.77:0/381881783 learned_addr learned my addr 172.21.15.77:0/381881783 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:52.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb5791640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb01033c0 0x7f1bb019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1baffff640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bb0102a00 0x7f1bb0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 -- 172.21.15.77:0/381881783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bb0102a00 msgr2=0x7f1bb0197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bb0102a00 0x7f1bb0197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 -- 172.21.15.77:0/381881783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb01033c0 msgr2=0x7f1bb019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb01033c0 0x7f1bb019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 -- 172.21.15.77:0/381881783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb019f610 con 0x7f1bb0101810 2024-03-16T09:39:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.239+0000 7f1bb4f90640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb0101810 0x7f1bb01979f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400d950 tx=0x7f1ba400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.240+0000 7f1badffb640 1 -- 172.21.15.77:0/381881783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4017020 con 0x7f1bb0101810 2024-03-16T09:39:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.240+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb019f900 con 0x7f1bb0101810 2024-03-16T09:39:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.240+0000 7f1badffb640 1 -- 172.21.15.77:0/381881783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ba4009c30 con 0x7f1bb0101810 2024-03-16T09:39:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.240+0000 7f1badffb640 1 -- 172.21.15.77:0/381881783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400f6b0 con 0x7f1bb0101810 2024-03-16T09:39:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.240+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb019fea0 con 0x7f1bb0101810 2024-03-16T09:39:52.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.241+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb0067d40 con 0x7f1bb0101810 2024-03-16T09:39:52.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.242+0000 7f1badffb640 1 -- 172.21.15.77:0/381881783 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1ba401a020 con 0x7f1bb0101810 2024-03-16T09:39:52.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.243+0000 7f1badffb640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ba00a2800 0x7f1ba00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.243+0000 7f1baffff640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ba00a2800 0x7f1ba00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.243+0000 7f1badffb640 1 -- 172.21.15.77:0/381881783 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1ba40ce690 con 0x7f1bb0101810 2024-03-16T09:39:52.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.243+0000 7f1baffff640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ba00a2800 0x7f1ba00a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0198f00 tx=0x7f1b9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:52.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.247+0000 7f1badffb640 1 -- 172.21.15.77:0/381881783 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ba4009de0 con 0x7f1bb0101810 2024-03-16T09:39:52.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.505+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bb0108b80 con 0x7f1bb0101810 2024-03-16T09:39:52.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.505+0000 7f1badffb640 1 -- 172.21.15.77:0/381881783 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7f1ba409bb30 con 0x7f1bb0101810 2024-03-16T09:39:52.510 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:52.510 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-16T09:39:52.511 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-16T09:39:52.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ba00a2800 msgr2=0x7f1ba00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ba00a2800 0x7f1ba00a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0198f00 tx=0x7f1b9c005f90 comp rx=0 tx=0).stop 2024-03-16T09:39:52.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb0101810 msgr2=0x7f1bb01979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb0101810 0x7f1bb01979f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400d950 tx=0x7f1ba400de10 comp rx=0 tx=0).stop 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 shutdown_connections 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1ba00a2800 0x7f1ba00a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb01033c0 0x7f1bb019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bb0102a00 0x7f1bb0197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 --2- 172.21.15.77:0/381881783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb0101810 0x7f1bb01979f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.507+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 >> 172.21.15.77:0/381881783 conn(0x7f1bb00fcd80 msgr2=0x7f1bb0105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.508+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 shutdown_connections 2024-03-16T09:39:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.508+0000 7f1bb71f4640 1 -- 172.21.15.77:0/381881783 wait complete. 2024-03-16T09:39:52.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:39:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 -- 172.21.15.77:0/3841200762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 msgr2=0x7fd940108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 --2- 172.21.15.77:0/3841200762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd940108320 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd938009ca0 tx=0x7fd93802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 -- 172.21.15.77:0/3841200762 shutdown_connections 2024-03-16T09:39:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 --2- 172.21.15.77:0/3841200762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd940108320 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 --2- 172.21.15.77:0/3841200762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd940101040 0x7fd9401014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 --2- 172.21.15.77:0/3841200762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9400ffe60 0x7fd940100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 -- 172.21.15.77:0/3841200762 >> 172.21.15.77:0/3841200762 conn(0x7fd9400fb420 msgr2=0x7fd9400fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 -- 172.21.15.77:0/3841200762 shutdown_connections 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.622+0000 7fd9485d3640 1 -- 172.21.15.77:0/3841200762 wait complete. 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 Processor -- start 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 -- start start 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9400ffe60 0x7fd9401969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd940101040 0x7fd940196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd94019df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd94010ba90 con 0x7fd940101a00 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd94010b910 con 0x7fd9400ffe60 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9485d3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd94010bc10 con 0x7fd940101040 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd94019df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd94019df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52852/0 (socket says 172.21.15.77:52852) 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 -- 172.21.15.77:0/2260736169 learned_addr learned my addr 172.21.15.77:0/2260736169 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd945b6e640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd940101040 0x7fd940196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 -- 172.21.15.77:0/2260736169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd940101040 msgr2=0x7fd940196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd940101040 0x7fd940196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 -- 172.21.15.77:0/2260736169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9400ffe60 msgr2=0x7fd9401969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9400ffe60 0x7fd9401969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 -- 172.21.15.77:0/2260736169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94019e5e0 con 0x7fd940101a00 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd945b6e640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd940101040 0x7fd940196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd946b70640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd94019df70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd93802c700 tx=0x7fd9380328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.623+0000 7fd9377fe640 1 -- 172.21.15.77:0/2260736169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938041020 con 0x7fd940101a00 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.624+0000 7fd9377fe640 1 -- 172.21.15.77:0/2260736169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd938003e20 con 0x7fd940101a00 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.624+0000 7fd9377fe640 1 -- 172.21.15.77:0/2260736169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938039040 con 0x7fd940101a00 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.624+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94019e870 con 0x7fd940101a00 2024-03-16T09:39:52.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.624+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd94019ede0 con 0x7fd940101a00 2024-03-16T09:39:52.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.624+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd940107610 con 0x7fd940101a00 2024-03-16T09:39:52.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.626+0000 7fd9377fe640 1 -- 172.21.15.77:0/2260736169 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd938044020 con 0x7fd940101a00 2024-03-16T09:39:52.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.626+0000 7fd9377fe640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.626+0000 7fd94636f640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.626+0000 7fd9377fe640 1 -- 172.21.15.77:0/2260736169 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd9380f0ea0 con 0x7fd940101a00 2024-03-16T09:39:52.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.627+0000 7fd94636f640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd940100bb0 tx=0x7fd93c006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:52.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.630+0000 7fd9377fe640 1 -- 172.21.15.77:0/2260736169 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd93804b020 con 0x7fd940101a00 2024-03-16T09:39:52.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.806+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd940062410 con 0x7fd940101a00 2024-03-16T09:39:52.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.807+0000 7fd9377fe640 1 -- 172.21.15.77:0/2260736169 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18468 (secure 0 0 0) 0x7fd9380be3e0 con 0x7fd940101a00 2024-03-16T09:39:52.811 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:52.811 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":82,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:07.686492+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:39:52.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd92c0a2800 msgr2=0x7fd92c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd940100bb0 tx=0x7fd93c006c90 comp rx=0 tx=0).stop 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 msgr2=0x7fd94019df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd94019df70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd93802c700 tx=0x7fd9380328a0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 shutdown_connections 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd940101a00 0x7fd94019df70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd940101040 0x7fd940196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 --2- 172.21.15.77:0/2260736169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9400ffe60 0x7fd9401969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 >> 172.21.15.77:0/2260736169 conn(0x7fd9400fb420 msgr2=0x7fd9401047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 shutdown_connections 2024-03-16T09:39:52.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.810+0000 7fd9485d3640 1 -- 172.21.15.77:0/2260736169 wait complete. 2024-03-16T09:39:52.824 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:39:52.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:39:52.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 -- 172.21.15.77:0/2355760717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 msgr2=0x7f8598101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2355760717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f8598101d60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8588009900 tx=0x7f858802bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 -- 172.21.15.77:0/2355760717 shutdown_connections 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2355760717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85981022a0 0x7f8598108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2355760717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f8598101d60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2355760717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85981006f0 0x7f8598100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 -- 172.21.15.77:0/2355760717 >> 172.21.15.77:0/2355760717 conn(0x7f85980fbc80 msgr2=0x7f85980fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 -- 172.21.15.77:0/2355760717 shutdown_connections 2024-03-16T09:39:52.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.927+0000 7f859fe7e640 1 -- 172.21.15.77:0/2355760717 wait complete. 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 Processor -- start 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 -- start start 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85981006f0 0x7f859806a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f859819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85981022a0 0x7f859819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f859810c220 con 0x7f85981018e0 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f859810c0a0 con 0x7f85981022a0 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859fe7e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f859810c3a0 con 0x7f85981006f0 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f859819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f859819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52868/0 (socket says 172.21.15.77:52868) 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 -- 172.21.15.77:0/2730587630 learned_addr learned my addr 172.21.15.77:0/2730587630 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859dc1a640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85981006f0 0x7f859806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859e41b640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85981022a0 0x7f859819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 -- 172.21.15.77:0/2730587630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85981006f0 msgr2=0x7f859806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85981006f0 0x7f859806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 -- 172.21.15.77:0/2730587630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85981022a0 msgr2=0x7f859819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85981022a0 0x7f859819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 -- 172.21.15.77:0/2730587630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859819f6c0 con 0x7f85981018e0 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.928+0000 7f859d419640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f859819c7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8588006010 tx=0x7f85880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.929+0000 7f858effd640 1 -- 172.21.15.77:0/2730587630 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588043020 con 0x7f85981018e0 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.929+0000 7f858effd640 1 -- 172.21.15.77:0/2730587630 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8588004aa0 con 0x7f85981018e0 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.929+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859819f950 con 0x7f85981018e0 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.929+0000 7f858effd640 1 -- 172.21.15.77:0/2730587630 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588003c60 con 0x7f85981018e0 2024-03-16T09:39:52.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.929+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f859819fea0 con 0x7f85981018e0 2024-03-16T09:39:52.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.930+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8598107b50 con 0x7f85981018e0 2024-03-16T09:39:52.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.931+0000 7f858effd640 1 -- 172.21.15.77:0/2730587630 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8588046020 con 0x7f85981018e0 2024-03-16T09:39:52.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.931+0000 7f858effd640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85840a2800 0x7f85840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:52.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.931+0000 7f859dc1a640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85840a2800 0x7f85840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:52.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.931+0000 7f858effd640 1 -- 172.21.15.77:0/2730587630 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f85880f08f0 con 0x7f85981018e0 2024-03-16T09:39:52.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.932+0000 7f859dc1a640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85840a2800 0x7f85840a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8594005fd0 tx=0x7f859400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:52.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:52.936+0000 7f858effd640 1 -- 172.21.15.77:0/2730587630 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f858804d020 con 0x7f85981018e0 2024-03-16T09:39:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.140+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f859802d3d0 con 0x7f85981018e0 2024-03-16T09:39:53.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.140+0000 7f858effd640 1 -- 172.21.15.77:0/2730587630 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f85880bde30 con 0x7f85981018e0 2024-03-16T09:39:53.143 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:39:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85840a2800 msgr2=0x7f85840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85840a2800 0x7f85840a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8594005fd0 tx=0x7f859400b390 comp rx=0 tx=0).stop 2024-03-16T09:39:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 msgr2=0x7f859819c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f859819c7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8588006010 tx=0x7f85880042f0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 shutdown_connections 2024-03-16T09:39:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85840a2800 0x7f85840a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85981022a0 0x7f859819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85981018e0 0x7f859819c7c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 --2- 172.21.15.77:0/2730587630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85981006f0 0x7f859806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 >> 172.21.15.77:0/2730587630 conn(0x7f85980fbc80 msgr2=0x7f8598104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 shutdown_connections 2024-03-16T09:39:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.142+0000 7f859fe7e640 1 -- 172.21.15.77:0/2730587630 wait complete. 2024-03-16T09:39:53.155 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/381881783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:52 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2260736169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:53.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T09:39:53.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/381881783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:52 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2260736169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:53.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/3587319702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1018f0 msgr2=0x7fbdbc101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/3587319702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1018f0 0x7fbdbc101d70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009900 tx=0x7fbdac02bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/3587319702 shutdown_connections 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/3587319702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdbc1022b0 0x7fbdbc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/3587319702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1018f0 0x7fbdbc101d70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/3587319702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdbc100700 0x7fbdbc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/3587319702 >> 172.21.15.77:0/3587319702 conn(0x7fbdbc0fbc70 msgr2=0x7fbdbc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/3587319702 shutdown_connections 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.260+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/3587319702 wait complete. 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 Processor -- start 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 -- start start 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdbc100700 0x7fbdbc104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdbc1018f0 0x7fbdbc102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1022b0 0x7fbdbc103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc10c230 con 0x7fbdbc1022b0 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc10c0b0 con 0x7fbdbc1018f0 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc3a0c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc10c3b0 con 0x7fbdbc100700 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1022b0 0x7fbdbc103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1022b0 0x7fbdbc103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52886/0 (socket says 172.21.15.77:52886) 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 -- 172.21.15.77:0/4250910854 learned_addr learned my addr 172.21.15.77:0/4250910854 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc17a8640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdbc100700 0x7fbdbc104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc0fa7640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdbc1018f0 0x7fbdbc102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 -- 172.21.15.77:0/4250910854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdbc100700 msgr2=0x7fbdbc104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdbc100700 0x7fbdbc104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 -- 172.21.15.77:0/4250910854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdbc1018f0 msgr2=0x7fbdbc102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdbc1018f0 0x7fbdbc102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.261+0000 7fbdc1fa9640 1 -- 172.21.15.77:0/4250910854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdbc103aa0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.262+0000 7fbdc0fa7640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdbc1018f0 0x7fbdbc102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:39:53.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.262+0000 7fbdc1fa9640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1022b0 0x7fbdbc103210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400eaa0 tx=0x7fbdb400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:53.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.262+0000 7fbdb27fc640 1 -- 172.21.15.77:0/4250910854 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb400cdb0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.262+0000 7fbdb27fc640 1 -- 172.21.15.77:0/4250910854 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdb4005020 con 0x7fbdbc1022b0 2024-03-16T09:39:53.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.262+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdbc1a1ba0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.262+0000 7fbdb27fc640 1 -- 172.21.15.77:0/4250910854 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb40116d0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.262+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdbc1a20a0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.263+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdbc1a22d0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.264+0000 7fbdb27fc640 1 -- 172.21.15.77:0/4250910854 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbdb40051d0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.265+0000 7fbdb27fc640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbda80a2800 0x7fbda80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.265+0000 7fbdc17a8640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbda80a2800 0x7fbda80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.265+0000 7fbdb27fc640 1 -- 172.21.15.77:0/4250910854 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbdb40ce7e0 con 0x7fbdbc1022b0 2024-03-16T09:39:53.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.265+0000 7fbdc17a8640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbda80a2800 0x7fbda80a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80058c0 tx=0x7fbdb800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:53.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.269+0000 7fbdb27fc640 1 -- 172.21.15.77:0/4250910854 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbdb4016030 con 0x7fbdbc1022b0 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/381881783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:52 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2260736169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:53.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.452+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbdbc1a2660 con 0x7fbdbc1022b0 2024-03-16T09:39:53.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.453+0000 7fbdb27fc640 1 -- 172.21.15.77:0/4250910854 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fbdb409bd20 con 0x7fbdbc1022b0 2024-03-16T09:39:53.456 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:53.456 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-16T09:39:53.456 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:39:53.456 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-16T09:39:53.456 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-16T09:39:53.457 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:39:53.457 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-16T09:39:53.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbda80a2800 msgr2=0x7fbda80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbda80a2800 0x7fbda80a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80058c0 tx=0x7fbdb800a2d0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1022b0 msgr2=0x7fbdbc103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1022b0 0x7fbdbc103210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400eaa0 tx=0x7fbdb400ef60 comp rx=0 tx=0).stop 2024-03-16T09:39:53.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 shutdown_connections 2024-03-16T09:39:53.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbda80a2800 0x7fbda80a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdbc1022b0 0x7fbdbc103210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdbc1018f0 0x7fbdbc102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 --2- 172.21.15.77:0/4250910854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdbc100700 0x7fbdbc104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 >> 172.21.15.77:0/4250910854 conn(0x7fbdbc0fbc70 msgr2=0x7fbdbc068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:53.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 shutdown_connections 2024-03-16T09:39:53.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.455+0000 7fbdc3a0c640 1 -- 172.21.15.77:0/4250910854 wait complete. 2024-03-16T09:39:53.469 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T09:39:53.470 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T09:39:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 -- 172.21.15.77:0/361456907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 msgr2=0x7f8b14101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 --2- 172.21.15.77:0/361456907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b14101d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009900 tx=0x7f8b0802bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 -- 172.21.15.77:0/361456907 shutdown_connections 2024-03-16T09:39:53.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 --2- 172.21.15.77:0/361456907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b14102290 0x7f8b14108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 --2- 172.21.15.77:0/361456907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b14101d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 --2- 172.21.15.77:0/361456907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b141006e0 0x7f8b14100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 -- 172.21.15.77:0/361456907 >> 172.21.15.77:0/361456907 conn(0x7f8b140fbc70 msgr2=0x7f8b140fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 -- 172.21.15.77:0/361456907 shutdown_connections 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 -- 172.21.15.77:0/361456907 wait complete. 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.574+0000 7f8b19d62640 1 Processor -- start 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b19d62640 1 -- start start 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b19d62640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b141006e0 0x7f8b140694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b19d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b1419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b19d62640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b14102290 0x7f8b1419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b19d62640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1410c1f0 con 0x7f8b141018d0 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b19d62640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1410c070 con 0x7f8b14102290 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b19d62640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1410c370 con 0x7f8b141006e0 2024-03-16T09:39:53.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b1419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b1419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52910/0 (socket says 172.21.15.77:52910) 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 -- 172.21.15.77:0/2647132495 learned_addr learned my addr 172.21.15.77:0/2647132495 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b137fe640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b141006e0 0x7f8b140694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b13fff640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b14102290 0x7f8b1419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 -- 172.21.15.77:0/2647132495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b141006e0 msgr2=0x7f8b140694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b141006e0 0x7f8b140694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 -- 172.21.15.77:0/2647132495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b14102290 msgr2=0x7f8b1419eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b14102290 0x7f8b1419eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 -- 172.21.15.77:0/2647132495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1419f6b0 con 0x7f8b141018d0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.575+0000 7f8b12ffd640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b1419c780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b08006010 tx=0x7f8b08005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.576+0000 7f8b10ff9640 1 -- 172.21.15.77:0/2647132495 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08043020 con 0x7f8b141018d0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.576+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1419f8e0 con 0x7f8b141018d0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.576+0000 7f8b10ff9640 1 -- 172.21.15.77:0/2647132495 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b080043d0 con 0x7f8b141018d0 2024-03-16T09:39:53.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.576+0000 7f8b10ff9640 1 -- 172.21.15.77:0/2647132495 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08003b80 con 0x7f8b141018d0 2024-03-16T09:39:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.576+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1419fe80 con 0x7f8b141018d0 2024-03-16T09:39:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.577+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b14107b20 con 0x7f8b141018d0 2024-03-16T09:39:53.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.578+0000 7f8b10ff9640 1 -- 172.21.15.77:0/2647132495 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8b08046020 con 0x7f8b141018d0 2024-03-16T09:39:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.579+0000 7f8b10ff9640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.579+0000 7f8b10ff9640 1 -- 172.21.15.77:0/2647132495 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8b080f0770 con 0x7f8b141018d0 2024-03-16T09:39:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.579+0000 7f8b137fe640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:53.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.579+0000 7f8b137fe640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b14101730 tx=0x7f8b04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:53.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.583+0000 7f8b10ff9640 1 -- 172.21.15.77:0/2647132495 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b0804d020 con 0x7f8b141018d0 2024-03-16T09:39:53.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:53.771+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8b1402d380 con 0x7f8b141018d0 2024-03-16T09:39:54.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.159+0000 7f8b10ff9640 1 -- 172.21.15.77:0/2647132495 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f8b080bdcb0 con 0x7f8b141018d0 2024-03-16T09:39:54.162 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-16T09:39:54.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8afc0a2800 msgr2=0x7f8afc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:54.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b14101730 tx=0x7f8b04005f90 comp rx=0 tx=0).stop 2024-03-16T09:39:54.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 msgr2=0x7f8b1419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:54.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b1419c780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b08006010 tx=0x7f8b08005e40 comp rx=0 tx=0).stop 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 shutdown_connections 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8afc0a2800 0x7f8afc0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b14102290 0x7f8b1419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b141018d0 0x7f8b1419c780 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 --2- 172.21.15.77:0/2647132495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b141006e0 0x7f8b140694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 >> 172.21.15.77:0/2647132495 conn(0x7f8b140fbc70 msgr2=0x7f8b14104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 shutdown_connections 2024-03-16T09:39:54.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.161+0000 7f8b19d62640 1 -- 172.21.15.77:0/2647132495 wait complete. 2024-03-16T09:39:54.179 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 -- 172.21.15.77:0/2399924313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58100250 msgr2=0x7fac58109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/2399924313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58100250 0x7fac58109e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fac50009ca0 tx=0x7fac5002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 -- 172.21.15.77:0/2399924313 shutdown_connections 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/2399924313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58100250 0x7fac58109e00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/2399924313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac580ff890 0x7fac580ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/2399924313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac58102e20 0x7fac58103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 -- 172.21.15.77:0/2399924313 >> 172.21.15.77:0/2399924313 conn(0x7fac580fb410 msgr2=0x7fac580fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 -- 172.21.15.77:0/2399924313 shutdown_connections 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.286+0000 7fac5f3e6640 1 -- 172.21.15.77:0/2399924313 wait complete. 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 Processor -- start 2024-03-16T09:39:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 -- start start 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac580ff890 0x7fac581979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac58100250 0x7fac58197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58102e20 0x7fac5819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac5810d320 con 0x7fac58102e20 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fac5810d1a0 con 0x7fac580ff890 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac5810d4a0 con 0x7fac58100250 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58102e20 0x7fac5819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58102e20 0x7fac5819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52932/0 (socket says 172.21.15.77:52932) 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 -- 172.21.15.77:0/551577154 learned_addr learned my addr 172.21.15.77:0/551577154 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5c981640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac58100250 0x7fac58197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d182640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac580ff890 0x7fac581979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 -- 172.21.15.77:0/551577154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac58100250 msgr2=0x7fac58197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac58100250 0x7fac58197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 -- 172.21.15.77:0/551577154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac580ff890 msgr2=0x7fac581979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac580ff890 0x7fac581979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 -- 172.21.15.77:0/551577154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5819f5e0 con 0x7fac58102e20 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5d983640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58102e20 0x7fac5819ef70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fac5002c700 tx=0x7fac500328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac4e7fc640 1 -- 172.21.15.77:0/551577154 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50041020 con 0x7fac58102e20 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac4e7fc640 1 -- 172.21.15.77:0/551577154 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac50003e20 con 0x7fac58102e20 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5819f870 con 0x7fac58102e20 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.287+0000 7fac4e7fc640 1 -- 172.21.15.77:0/551577154 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50039040 con 0x7fac58102e20 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.288+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5819fdb0 con 0x7fac58102e20 2024-03-16T09:39:54.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.288+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac58108d00 con 0x7fac58102e20 2024-03-16T09:39:54.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.290+0000 7fac4e7fc640 1 -- 172.21.15.77:0/551577154 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fac50044020 con 0x7fac58102e20 2024-03-16T09:39:54.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.290+0000 7fac4e7fc640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac440a2880 0x7fac440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:54.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.290+0000 7fac5d182640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac440a2880 0x7fac440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:54.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.290+0000 7fac4e7fc640 1 -- 172.21.15.77:0/551577154 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fac500f0ef0 con 0x7fac58102e20 2024-03-16T09:39:54.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.291+0000 7fac5d182640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac440a2880 0x7fac440a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fac54005fd0 tx=0x7fac54005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:54.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.294+0000 7fac4e7fc640 1 -- 172.21.15.77:0/551577154 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac5004b020 con 0x7fac58102e20 2024-03-16T09:39:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:54 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2730587630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:54 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:54.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:54 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4250910854' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T09:39:54.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:54 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2730587630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:54.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:54.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:54.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:54.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:54 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:54.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:54 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4250910854' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T09:39:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:54.478+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fac5802d1b0 con 0x7fac58102e20 2024-03-16T09:39:54.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:54 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2730587630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:39:54.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:54.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:54.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:54.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:54 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:39:54.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:54 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4250910854' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T09:39:55.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.162+0000 7fac4e7fc640 1 -- 172.21.15.77:0/551577154 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7fac500be2c0 con 0x7fac58102e20 2024-03-16T09:39:55.165 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac440a2880 msgr2=0x7fac440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac440a2880 0x7fac440a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fac54005fd0 tx=0x7fac54005850 comp rx=0 tx=0).stop 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58102e20 msgr2=0x7fac5819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58102e20 0x7fac5819ef70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fac5002c700 tx=0x7fac500328a0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 shutdown_connections 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac440a2880 0x7fac440a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac58102e20 0x7fac5819ef70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac58100250 0x7fac58197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 --2- 172.21.15.77:0/551577154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac580ff890 0x7fac581979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 >> 172.21.15.77:0/551577154 conn(0x7fac580fb410 msgr2=0x7fac58105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:55.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 shutdown_connections 2024-03-16T09:39:55.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.165+0000 7fac5f3e6640 1 -- 172.21.15.77:0/551577154 wait complete. 2024-03-16T09:39:55.186 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 -- 172.21.15.77:0/2790447591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241029e0 msgr2=0x7f7a24102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/2790447591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241029e0 0x7f7a24102e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009900 tx=0x7f7a1802bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 -- 172.21.15.77:0/2790447591 shutdown_connections 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/2790447591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a241033a0 0x7f7a24109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/2790447591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241029e0 0x7f7a24102e60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/2790447591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a241017f0 0x7f7a24101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 -- 172.21.15.77:0/2790447591 >> 172.21.15.77:0/2790447591 conn(0x7f7a240fcd80 msgr2=0x7f7a240ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 -- 172.21.15.77:0/2790447591 shutdown_connections 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 -- 172.21.15.77:0/2790447591 wait complete. 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 Processor -- start 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.289+0000 7f7a28ef7640 1 -- start start 2024-03-16T09:39:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a241017f0 0x7f7a24197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a241029e0 0x7f7a24197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241033a0 0x7f7a2419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2410d300 con 0x7f7a241033a0 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2410d180 con 0x7f7a241017f0 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2410d480 con 0x7f7a241029e0 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241033a0 0x7f7a2419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241033a0 0x7f7a2419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52946/0 (socket says 172.21.15.77:52946) 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 -- 172.21.15.77:0/93389797 learned_addr learned my addr 172.21.15.77:0/93389797 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a2259c640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a241017f0 0x7f7a24197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a21d9b640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a241029e0 0x7f7a24197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 -- 172.21.15.77:0/93389797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a241029e0 msgr2=0x7f7a24197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a241029e0 0x7f7a24197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 -- 172.21.15.77:0/93389797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a241017f0 msgr2=0x7f7a24197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:55.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a241017f0 0x7f7a24197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 -- 172.21.15.77:0/93389797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2419f5b0 con 0x7f7a241033a0 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a2259c640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a241017f0 0x7f7a24197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a22d9d640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241033a0 0x7f7a2419ef40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00eaa0 tx=0x7f7a1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a137fe640 1 -- 172.21.15.77:0/93389797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00cdb0 con 0x7f7a241033a0 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2419f8a0 con 0x7f7a241033a0 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a137fe640 1 -- 172.21.15.77:0/93389797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a1c005020 con 0x7f7a241033a0 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a137fe640 1 -- 172.21.15.77:0/93389797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0116d0 con 0x7f7a241033a0 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.290+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2419fda0 con 0x7f7a241033a0 2024-03-16T09:39:55.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.291+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a24108be0 con 0x7f7a241033a0 2024-03-16T09:39:55.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.293+0000 7f7a137fe640 1 -- 172.21.15.77:0/93389797 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7a1c0051d0 con 0x7f7a241033a0 2024-03-16T09:39:55.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.293+0000 7f7a137fe640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a0c0a2800 0x7f7a0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:55.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.293+0000 7f7a2259c640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a0c0a2800 0x7f7a0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:55.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.293+0000 7f7a137fe640 1 -- 172.21.15.77:0/93389797 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f7a1c0cef50 con 0x7f7a241033a0 2024-03-16T09:39:55.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.294+0000 7f7a2259c640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a0c0a2800 0x7f7a0c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7a24102840 tx=0x7f7a1400a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:55.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.297+0000 7f7a137fe640 1 -- 172.21.15.77:0/93389797 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a1c016030 con 0x7f7a241033a0 2024-03-16T09:39:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2647132495' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2647132495' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/551577154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2647132495' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2647132495' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/551577154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:55.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:55.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2647132495' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2647132495' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/551577154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:39:55.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:39:55.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:39:55.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:55.509+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7a2402d1b0 con 0x7f7a241033a0 2024-03-16T09:39:56.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.171+0000 7f7a137fe640 1 -- 172.21.15.77:0/93389797 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f7a1c09c1a0 con 0x7f7a241033a0 2024-03-16T09:39:56.174 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a0c0a2800 msgr2=0x7f7a0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a0c0a2800 0x7f7a0c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7a24102840 tx=0x7f7a1400a5e0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241033a0 msgr2=0x7f7a2419ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241033a0 0x7f7a2419ef40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00eaa0 tx=0x7f7a1c00ef60 comp rx=0 tx=0).stop 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 shutdown_connections 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a0c0a2800 0x7f7a0c0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a241033a0 0x7f7a2419ef40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a241029e0 0x7f7a24197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 --2- 172.21.15.77:0/93389797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a241017f0 0x7f7a24197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 >> 172.21.15.77:0/93389797 conn(0x7f7a240fcd80 msgr2=0x7f7a24105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:56.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 shutdown_connections 2024-03-16T09:39:56.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.176+0000 7f7a28ef7640 1 -- 172.21.15.77:0/93389797 wait complete. 2024-03-16T09:39:56.202 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:39:56.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 -- 172.21.15.77:0/1408472143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80107580 msgr2=0x7f1a80109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 --2- 172.21.15.77:0/1408472143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80107580 0x7f1a80109970 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009ca0 tx=0x7f1a7802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 -- 172.21.15.77:0/1408472143 shutdown_connections 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 --2- 172.21.15.77:0/1408472143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80107580 0x7f1a80109970 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 --2- 172.21.15.77:0/1408472143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a80104c50 0x7f1a80107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 --2- 172.21.15.77:0/1408472143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a80102320 0x7f1a80104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 -- 172.21.15.77:0/1408472143 >> 172.21.15.77:0/1408472143 conn(0x7f1a800fbc70 msgr2=0x7f1a800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 -- 172.21.15.77:0/1408472143 shutdown_connections 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.305+0000 7f1a867f4640 1 -- 172.21.15.77:0/1408472143 wait complete. 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 Processor -- start 2024-03-16T09:39:56.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 -- start start 2024-03-16T09:39:56.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80102320 0x7f1a80197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a80104c50 0x7f1a80197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a80107580 0x7f1a8019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8010d350 con 0x7f1a80102320 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8010d1d0 con 0x7f1a80104c50 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a867f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8010d4d0 con 0x7f1a80107580 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a7ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80102320 0x7f1a80197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a7ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80102320 0x7f1a80197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59514/0 (socket says 172.21.15.77:59514) 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.306+0000 7f1a7ffff640 1 -- 172.21.15.77:0/316599108 learned_addr learned my addr 172.21.15.77:0/316599108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7f7fe640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a80104c50 0x7f1a80197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a84d91640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a80107580 0x7f1a8019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7ffff640 1 -- 172.21.15.77:0/316599108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a80107580 msgr2=0x7f1a8019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7ffff640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a80107580 0x7f1a8019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7ffff640 1 -- 172.21.15.77:0/316599108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a80104c50 msgr2=0x7f1a80197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7ffff640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a80104c50 0x7f1a80197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7ffff640 1 -- 172.21.15.77:0/316599108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8019f680 con 0x7f1a80102320 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7ffff640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80102320 0x7f1a80197a60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000d770 tx=0x7f1a7000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:56.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7d7fa640 1 -- 172.21.15.77:0/316599108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70017070 con 0x7f1a80102320 2024-03-16T09:39:56.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8019f970 con 0x7f1a80102320 2024-03-16T09:39:56.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7d7fa640 1 -- 172.21.15.77:0/316599108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a700054d0 con 0x7f1a80102320 2024-03-16T09:39:56.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a7d7fa640 1 -- 172.21.15.77:0/316599108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70004cf0 con 0x7f1a80102320 2024-03-16T09:39:56.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.307+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8019fec0 con 0x7f1a80102320 2024-03-16T09:39:56.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.308+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a800685d0 con 0x7f1a80102320 2024-03-16T09:39:56.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.309+0000 7f1a7d7fa640 1 -- 172.21.15.77:0/316599108 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1a70004e90 con 0x7f1a80102320 2024-03-16T09:39:56.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.310+0000 7f1a7d7fa640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a680a2800 0x7f1a680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.310+0000 7f1a7f7fe640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a680a2800 0x7f1a680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.310+0000 7f1a7d7fa640 1 -- 172.21.15.77:0/316599108 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1a700ce2c0 con 0x7f1a80102320 2024-03-16T09:39:56.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.310+0000 7f1a7f7fe640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a680a2800 0x7f1a680a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a80198f70 tx=0x7f1a7400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:56.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.315+0000 7f1a7d7fa640 1 -- 172.21.15.77:0/316599108 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a7009ae60 con 0x7f1a80102320 2024-03-16T09:39:56.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:56 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/551577154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T09:39:56.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:56 smithi119 ceph-mon[35632]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T09:39:56.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:56 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/93389797' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T09:39:56.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:56 smithi119 ceph-mon[35632]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T09:39:56.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:56 smithi119 ceph-mon[35632]: pgmap v283: 97 pgs: 7 creating+peering, 1 active+clean, 89 unknown; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:56.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:56 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/551577154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T09:39:56.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:56 smithi077 ceph-mon[32171]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T09:39:56.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:56 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/93389797' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T09:39:56.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:56 smithi077 ceph-mon[32171]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T09:39:56.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:56 smithi077 ceph-mon[32171]: pgmap v283: 97 pgs: 7 creating+peering, 1 active+clean, 89 unknown; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:56.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.517+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a800623e0 con 0x7f1a80102320 2024-03-16T09:39:56.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.519+0000 7f1a7d7fa640 1 -- 172.21.15.77:0/316599108 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22244 (secure 0 0 0) 0x7f1a7009ef00 con 0x7f1a80102320 2024-03-16T09:39:56.521 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:56.521 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:55.509457+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a680a2800 msgr2=0x7f1a680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a680a2800 0x7f1a680a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a80198f70 tx=0x7f1a7400a2d0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80102320 msgr2=0x7f1a80197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80102320 0x7f1a80197a60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000d770 tx=0x7f1a7000dc30 comp rx=0 tx=0).stop 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 shutdown_connections 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a680a2800 0x7f1a680a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a80107580 0x7f1a8019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a80104c50 0x7f1a80197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 --2- 172.21.15.77:0/316599108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a80102320 0x7f1a80197a60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 >> 172.21.15.77:0/316599108 conn(0x7f1a800fbc70 msgr2=0x7f1a80102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 shutdown_connections 2024-03-16T09:39:56.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.521+0000 7f1a867f4640 1 -- 172.21.15.77:0/316599108 wait complete. 2024-03-16T09:39:56.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:39:56.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:56 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/551577154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T09:39:56.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:56 smithi129 ceph-mon[35841]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T09:39:56.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:56 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/93389797' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T09:39:56.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:56 smithi129 ceph-mon[35841]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T09:39:56.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:56 smithi129 ceph-mon[35841]: pgmap v283: 97 pgs: 7 creating+peering, 1 active+clean, 89 unknown; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:39:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1916934841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a38101800 msgr2=0x7f1a38101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1916934841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a38101800 0x7f1a38101c20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009900 tx=0x7f1a3402bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1916934841 shutdown_connections 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1916934841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a381033b0 0x7f1a38109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1916934841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a381029f0 0x7f1a38102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1916934841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a38101800 0x7f1a38101c20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1916934841 >> 172.21.15.77:0/1916934841 conn(0x7f1a380fcd90 msgr2=0x7f1a380ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1916934841 shutdown_connections 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1916934841 wait complete. 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.633+0000 7f1a3ff1b640 1 Processor -- start 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 -- start start 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a38101800 0x7f1a38197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a381029f0 0x7f1a38197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a381033b0 0x7f1a3819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810d300 con 0x7f1a381029f0 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810d180 con 0x7f1a38101800 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810d480 con 0x7f1a381033b0 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a381029f0 0x7f1a38197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a381029f0 0x7f1a38197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59546/0 (socket says 172.21.15.77:59546) 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 -- 172.21.15.77:0/1823280724 learned_addr learned my addr 172.21.15.77:0/1823280724 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3dcb7640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a38101800 0x7f1a38197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 -- 172.21.15.77:0/1823280724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a381033b0 msgr2=0x7f1a3819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a381033b0 0x7f1a3819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 -- 172.21.15.77:0/1823280724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a38101800 msgr2=0x7f1a38197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a38101800 0x7f1a38197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 -- 172.21.15.77:0/1823280724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3819f560 con 0x7f1a381029f0 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3d4b6640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a381029f0 0x7f1a38197e80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800c920 tx=0x7f1a2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a2effd640 1 -- 172.21.15.77:0/1823280724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28007c70 con 0x7f1a381029f0 2024-03-16T09:39:56.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3819f850 con 0x7f1a381029f0 2024-03-16T09:39:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a2effd640 1 -- 172.21.15.77:0/1823280724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a28007e10 con 0x7f1a381029f0 2024-03-16T09:39:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a2effd640 1 -- 172.21.15.77:0/1823280724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28018820 con 0x7f1a381029f0 2024-03-16T09:39:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.634+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a3819fda0 con 0x7f1a381029f0 2024-03-16T09:39:56.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.635+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a38108b20 con 0x7f1a381029f0 2024-03-16T09:39:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.636+0000 7f1a2effd640 1 -- 172.21.15.77:0/1823280724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1a280189c0 con 0x7f1a381029f0 2024-03-16T09:39:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.637+0000 7f1a2effd640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a240a2800 0x7f1a240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.637+0000 7f1a3dcb7640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a240a2800 0x7f1a240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.637+0000 7f1a2effd640 1 -- 172.21.15.77:0/1823280724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1a280ced50 con 0x7f1a381029f0 2024-03-16T09:39:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.637+0000 7f1a3dcb7640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a240a2800 0x7f1a240a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009230 tx=0x7f1a34005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.640+0000 7f1a2effd640 1 -- 172.21.15.77:0/1823280724 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a28014030 con 0x7f1a381029f0 2024-03-16T09:39:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.816+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a3802d3b0 con 0x7f1a381029f0 2024-03-16T09:39:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.817+0000 7f1a2effd640 1 -- 172.21.15.77:0/1823280724 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22244 (secure 0 0 0) 0x7f1a2809bfd0 con 0x7f1a381029f0 2024-03-16T09:39:56.819 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:39:56.819 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:55.509457+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:39:56.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a240a2800 msgr2=0x7f1a240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a240a2800 0x7f1a240a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009230 tx=0x7f1a34005fd0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a381029f0 msgr2=0x7f1a38197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a381029f0 0x7f1a38197e80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800c920 tx=0x7f1a2800cde0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 shutdown_connections 2024-03-16T09:39:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a240a2800 0x7f1a240a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a381033b0 0x7f1a3819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a381029f0 0x7f1a38197e80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 --2- 172.21.15.77:0/1823280724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a38101800 0x7f1a38197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 >> 172.21.15.77:0/1823280724 conn(0x7f1a380fcd90 msgr2=0x7f1a38105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 shutdown_connections 2024-03-16T09:39:56.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.819+0000 7f1a3ff1b640 1 -- 172.21.15.77:0/1823280724 wait complete. 2024-03-16T09:39:56.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 -- 172.21.15.77:0/1047357918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 msgr2=0x7ff7b4109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/1047357918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b4109910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009ca0 tx=0x7ff7ac02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 -- 172.21.15.77:0/1047357918 shutdown_connections 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/1047357918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b4109910 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/1047357918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7b4104bf0 0x7ff7b4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/1047357918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7b41022c0 0x7ff7b41046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 -- 172.21.15.77:0/1047357918 >> 172.21.15.77:0/1047357918 conn(0x7ff7b40fbc70 msgr2=0x7ff7b40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 -- 172.21.15.77:0/1047357918 shutdown_connections 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 -- 172.21.15.77:0/1047357918 wait complete. 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 Processor -- start 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.935+0000 7ff7b9c36640 1 -- start start 2024-03-16T09:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7b41022c0 0x7ff7b41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7b4104bf0 0x7ff7b4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b410d380 con 0x7ff7b4107520 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b410d200 con 0x7ff7b4104bf0 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b410d500 con 0x7ff7b41022c0 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59566/0 (socket says 172.21.15.77:59566) 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 -- 172.21.15.77:0/3854028648 learned_addr learned my addr 172.21.15.77:0/3854028648 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b2ffd640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7b4104bf0 0x7ff7b4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 -- 172.21.15.77:0/3854028648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7b41022c0 msgr2=0x7ff7b41979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7b41022c0 0x7ff7b41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 -- 172.21.15.77:0/3854028648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7b4104bf0 msgr2=0x7ff7b4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7b4104bf0 0x7ff7b4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 -- 172.21.15.77:0/3854028648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b419f5f0 con 0x7ff7b4107520 2024-03-16T09:39:56.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b2ffd640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7b4104bf0 0x7ff7b4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:39:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b3fff640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b419ef80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac037040 tx=0x7ff7ac0328f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b0ff9640 1 -- 172.21.15.77:0/3854028648 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac041020 con 0x7ff7b4107520 2024-03-16T09:39:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7b419f880 con 0x7ff7b4107520 2024-03-16T09:39:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b0ff9640 1 -- 172.21.15.77:0/3854028648 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7ac003e20 con 0x7ff7b4107520 2024-03-16T09:39:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b0ff9640 1 -- 172.21.15.77:0/3854028648 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac039040 con 0x7ff7b4107520 2024-03-16T09:39:56.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.936+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7b419fe40 con 0x7ff7b4107520 2024-03-16T09:39:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.937+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7b40fd760 con 0x7ff7b4107520 2024-03-16T09:39:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.938+0000 7ff7b0ff9640 1 -- 172.21.15.77:0/3854028648 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff7ac044020 con 0x7ff7b4107520 2024-03-16T09:39:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.939+0000 7ff7b0ff9640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff79c0a2800 0x7ff79c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.939+0000 7ff7b37fe640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff79c0a2800 0x7ff79c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.939+0000 7ff7b0ff9640 1 -- 172.21.15.77:0/3854028648 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff7ac0f12e0 con 0x7ff7b4107520 2024-03-16T09:39:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.939+0000 7ff7b37fe640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff79c0a2800 0x7ff79c0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009710 tx=0x7ff7a4006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:56.942+0000 7ff7b0ff9640 1 -- 172.21.15.77:0/3854028648 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7ac04b020 con 0x7ff7b4107520 2024-03-16T09:39:57.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.153+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff7b402d1b0 con 0x7ff7b4107520 2024-03-16T09:39:57.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.173+0000 7ff7b0ff9640 1 -- 172.21.15.77:0/3854028648 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7ff7ac0098b0 con 0x7ff7b4107520 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff79c0a2800 msgr2=0x7ff79c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff79c0a2800 0x7ff79c0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009710 tx=0x7ff7a4006c90 comp rx=0 tx=0).stop 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 msgr2=0x7ff7b419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b419ef80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac037040 tx=0x7ff7ac0328f0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 shutdown_connections 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff79c0a2800 0x7ff79c0a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7b4107520 0x7ff7b419ef80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7b4104bf0 0x7ff7b4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 --2- 172.21.15.77:0/3854028648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7b41022c0 0x7ff7b41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 >> 172.21.15.77:0/3854028648 conn(0x7ff7b40fbc70 msgr2=0x7ff7b4102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:57.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 shutdown_connections 2024-03-16T09:39:57.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.175+0000 7ff7b9c36640 1 -- 172.21.15.77:0/3854028648 wait complete. 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/93389797' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/316599108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1823280724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:57.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3854028648' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T09:39:57.189 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-16T09:39:57.189 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-16T09:39:57.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 -- 172.21.15.77:0/2321907766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b0102290 msgr2=0x7fa1b0108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2321907766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b0102290 0x7fa1b0108b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009ca0 tx=0x7fa1a002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 -- 172.21.15.77:0/2321907766 shutdown_connections 2024-03-16T09:39:57.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2321907766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b0102290 0x7fa1b0108b20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2321907766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1b01018d0 0x7fa1b0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2321907766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1b01006e0 0x7fa1b0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 -- 172.21.15.77:0/2321907766 >> 172.21.15.77:0/2321907766 conn(0x7fa1b00fbc70 msgr2=0x7fa1b00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 -- 172.21.15.77:0/2321907766 shutdown_connections 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 -- 172.21.15.77:0/2321907766 wait complete. 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 Processor -- start 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.292+0000 7fa1b811d640 1 -- start start 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1b01006e0 0x7fa1b00694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b01018d0 0x7fa1b019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1b0102290 0x7fa1b019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b010c1f0 con 0x7fa1b01018d0 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b010c070 con 0x7fa1b01006e0 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b010c370 con 0x7fa1b0102290 2024-03-16T09:39:57.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b01018d0 0x7fa1b019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b01018d0 0x7fa1b019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59582/0 (socket says 172.21.15.77:59582) 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 -- 172.21.15.77:0/2939398000 learned_addr learned my addr 172.21.15.77:0/2939398000 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b5eb9640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1b01006e0 0x7fa1b00694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b66ba640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1b0102290 0x7fa1b019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 -- 172.21.15.77:0/2939398000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1b0102290 msgr2=0x7fa1b019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1b0102290 0x7fa1b019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 -- 172.21.15.77:0/2939398000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1b01006e0 msgr2=0x7fa1b00694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1b01006e0 0x7fa1b00694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 -- 172.21.15.77:0/2939398000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b019f560 con 0x7fa1b01018d0 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b56b8640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b01018d0 0x7fa1b019c660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400e9f0 tx=0x7fa1a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:57.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1aeffd640 1 -- 172.21.15.77:0/2939398000 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400cd80 con 0x7fa1b01018d0 2024-03-16T09:39:57.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1aeffd640 1 -- 172.21.15.77:0/2939398000 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1a40108e0 con 0x7fa1b01018d0 2024-03-16T09:39:57.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b019f850 con 0x7fa1b01018d0 2024-03-16T09:39:57.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1aeffd640 1 -- 172.21.15.77:0/2939398000 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4018920 con 0x7fa1b01018d0 2024-03-16T09:39:57.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.293+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b019fdf0 con 0x7fa1b01018d0 2024-03-16T09:39:57.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.294+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b01079a0 con 0x7fa1b01018d0 2024-03-16T09:39:57.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.295+0000 7fa1aeffd640 1 -- 172.21.15.77:0/2939398000 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa1a4010a90 con 0x7fa1b01018d0 2024-03-16T09:39:57.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.296+0000 7fa1aeffd640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa1980a2880 0x7fa1980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.296+0000 7fa1b5eb9640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa1980a2880 0x7fa1980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.296+0000 7fa1aeffd640 1 -- 172.21.15.77:0/2939398000 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa1a40cec60 con 0x7fa1b01018d0 2024-03-16T09:39:57.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.296+0000 7fa1b5eb9640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa1980a2880 0x7fa1980a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0101730 tx=0x7fa1a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:57.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.300+0000 7fa1aeffd640 1 -- 172.21.15.77:0/2939398000 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa1a409be50 con 0x7fa1b01018d0 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/93389797' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/316599108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1823280724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:57.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3854028648' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T09:39:57.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.513+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fa1b002d3d0 con 0x7fa1b01018d0 2024-03-16T09:39:57.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T09:39:57.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T09:39:57.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/93389797' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T09:39:57.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T09:39:57.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T09:39:57.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/316599108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:57.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1823280724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:39:57.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3854028648' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T09:39:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.703+0000 7fa1aeffd640 1 -- 172.21.15.77:0/2939398000 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7fa1a409b750 con 0x7fa1b01018d0 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa1980a2880 msgr2=0x7fa1980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa1980a2880 0x7fa1980a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0101730 tx=0x7fa1a8005f90 comp rx=0 tx=0).stop 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b01018d0 msgr2=0x7fa1b019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b01018d0 0x7fa1b019c660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400e9f0 tx=0x7fa1a400eeb0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 shutdown_connections 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa1980a2880 0x7fa1980a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1b0102290 0x7fa1b019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b01018d0 0x7fa1b019c660 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 --2- 172.21.15.77:0/2939398000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1b01006e0 0x7fa1b00694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 >> 172.21.15.77:0/2939398000 conn(0x7fa1b00fbc70 msgr2=0x7fa1b0104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:57.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 shutdown_connections 2024-03-16T09:39:57.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.705+0000 7fa1b811d640 1 -- 172.21.15.77:0/2939398000 wait complete. 2024-03-16T09:39:57.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-16T09:39:57.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 -- 172.21.15.77:0/952001960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 msgr2=0x7f2b44108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/952001960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b44108b20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009900 tx=0x7f2b4002bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 -- 172.21.15.77:0/952001960 shutdown_connections 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/952001960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b44108b20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/952001960 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b441018d0 0x7f2b44101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/952001960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b441006e0 0x7f2b44100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 -- 172.21.15.77:0/952001960 >> 172.21.15.77:0/952001960 conn(0x7f2b440fbc70 msgr2=0x7f2b440fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 -- 172.21.15.77:0/952001960 shutdown_connections 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.825+0000 7f2b4b53b640 1 -- 172.21.15.77:0/952001960 wait complete. 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 Processor -- start 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 -- start start 2024-03-16T09:39:57.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b441006e0 0x7f2b44197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b441018d0 0x7f2b44197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b4419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4410d330 con 0x7f2b44102290 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4410d1b0 con 0x7f2b441006e0 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b4b53b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4410d4b0 con 0x7f2b441018d0 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b4419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b4419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59596/0 (socket says 172.21.15.77:59596) 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 -- 172.21.15.77:0/343704581 learned_addr learned my addr 172.21.15.77:0/343704581 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b492d7640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b441006e0 0x7f2b44197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b48ad6640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b441018d0 0x7f2b44197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 -- 172.21.15.77:0/343704581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b441018d0 msgr2=0x7f2b44197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b441018d0 0x7f2b44197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 -- 172.21.15.77:0/343704581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b441006e0 msgr2=0x7f2b44197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:57.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b441006e0 0x7f2b44197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.826+0000 7f2b49ad8640 1 -- 172.21.15.77:0/343704581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4419f630 con 0x7f2b44102290 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.827+0000 7f2b49ad8640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b4419efc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009230 tx=0x7f2b400040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.827+0000 7f2b3a7fc640 1 -- 172.21.15.77:0/343704581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40043020 con 0x7f2b44102290 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.827+0000 7f2b3a7fc640 1 -- 172.21.15.77:0/343704581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b400044c0 con 0x7f2b44102290 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.827+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4419f8c0 con 0x7f2b44102290 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.827+0000 7f2b3a7fc640 1 -- 172.21.15.77:0/343704581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40003bc0 con 0x7f2b44102290 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.827+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4419fe10 con 0x7f2b44102290 2024-03-16T09:39:57.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.828+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b44107a50 con 0x7f2b44102290 2024-03-16T09:39:57.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.829+0000 7f2b3a7fc640 1 -- 172.21.15.77:0/343704581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2b40046020 con 0x7f2b44102290 2024-03-16T09:39:57.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.829+0000 7f2b3a7fc640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2b300a2800 0x7f2b300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:57.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.829+0000 7f2b492d7640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2b300a2800 0x7f2b300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:57.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.830+0000 7f2b3a7fc640 1 -- 172.21.15.77:0/343704581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2b400f0f80 con 0x7f2b44102290 2024-03-16T09:39:57.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.830+0000 7f2b492d7640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2b300a2800 0x7f2b300a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b44101730 tx=0x7f2b3c009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:57.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:57.833+0000 7f2b3a7fc640 1 -- 172.21.15.77:0/343704581 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b4004d020 con 0x7f2b44102290 2024-03-16T09:39:58.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.044+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f2b4402d330 con 0x7f2b44102290 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3854028648' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] up:active 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2939398000' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2939398000' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T09:39:58.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-16T09:39:58.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-16T09:39:58.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: pgmap v286: 97 pgs: 17 creating+peering, 43 active+clean, 37 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-16T09:39:58.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/343704581' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3854028648' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] up:active 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2939398000' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2939398000' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T09:39:58.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-16T09:39:58.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-16T09:39:58.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: pgmap v286: 97 pgs: 17 creating+peering, 43 active+clean, 37 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-16T09:39:58.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/343704581' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3854028648' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] up:active 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2939398000' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2939398000' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-16T09:39:58.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-16T09:39:58.552 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: pgmap v286: 97 pgs: 17 creating+peering, 43 active+clean, 37 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-16T09:39:58.552 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/343704581' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T09:39:58.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.713+0000 7f2b3a7fc640 1 -- 172.21.15.77:0/343704581 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f2b400be170 con 0x7f2b44102290 2024-03-16T09:39:58.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.715+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2b300a2800 msgr2=0x7f2b300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:58.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2b300a2800 0x7f2b300a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b44101730 tx=0x7f2b3c009680 comp rx=0 tx=0).stop 2024-03-16T09:39:58.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 msgr2=0x7f2b4419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:58.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b4419efc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009230 tx=0x7f2b400040c0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 shutdown_connections 2024-03-16T09:39:58.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2b300a2800 0x7f2b300a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b44102290 0x7f2b4419efc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b441018d0 0x7f2b44197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 --2- 172.21.15.77:0/343704581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b441006e0 0x7f2b44197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 >> 172.21.15.77:0/343704581 conn(0x7f2b440fbc70 msgr2=0x7f2b44104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:58.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 shutdown_connections 2024-03-16T09:39:58.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.716+0000 7f2b4b53b640 1 -- 172.21.15.77:0/343704581 wait complete. 2024-03-16T09:39:58.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T09:39:58.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.836+0000 7f3249bb3640 1 -- 172.21.15.77:0/2902777060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 msgr2=0x7f3244108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:58.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.836+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2902777060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f3244108b20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f323c009ca0 tx=0x7f323c02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- 172.21.15.77:0/2902777060 shutdown_connections 2024-03-16T09:39:58.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2902777060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f3244108b20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2902777060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32441018d0 0x7f3244101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2902777060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32441006e0 0x7f3244100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- 172.21.15.77:0/2902777060 >> 172.21.15.77:0/2902777060 conn(0x7f32440fbc70 msgr2=0x7f32440fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- 172.21.15.77:0/2902777060 shutdown_connections 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- 172.21.15.77:0/2902777060 wait complete. 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 Processor -- start 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- start start 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32441006e0 0x7f3244197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32441018d0 0x7f3244197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f324419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f324410d2f0 con 0x7f3244102290 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f324410d170 con 0x7f32441006e0 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.837+0000 7f3249bb3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f324410d470 con 0x7f32441018d0 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f324419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f32437fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32441006e0 0x7f3244197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:58.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f324419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59600/0 (socket says 172.21.15.77:59600) 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 -- 172.21.15.77:0/2513502169 learned_addr learned my addr 172.21.15.77:0/2513502169 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3242ffd640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32441018d0 0x7f3244197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 -- 172.21.15.77:0/2513502169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32441018d0 msgr2=0x7f3244197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32441018d0 0x7f3244197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 -- 172.21.15.77:0/2513502169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32441006e0 msgr2=0x7f3244197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32441006e0 0x7f3244197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 -- 172.21.15.77:0/2513502169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324419f5b0 con 0x7f3244102290 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f32437fe640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32441006e0 0x7f3244197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3243fff640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f324419ef40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f323c02c700 tx=0x7f323c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3240ff9640 1 -- 172.21.15.77:0/2513502169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c041020 con 0x7f3244102290 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3240ff9640 1 -- 172.21.15.77:0/2513502169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f323c003e20 con 0x7f3244102290 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324419f840 con 0x7f3244102290 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3240ff9640 1 -- 172.21.15.77:0/2513502169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c039040 con 0x7f3244102290 2024-03-16T09:39:58.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.838+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324419fd30 con 0x7f3244102290 2024-03-16T09:39:58.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.839+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3244107a00 con 0x7f3244102290 2024-03-16T09:39:58.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.840+0000 7f3240ff9640 1 -- 172.21.15.77:0/2513502169 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f323c044020 con 0x7f3244102290 2024-03-16T09:39:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.841+0000 7f3240ff9640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f322c0a2800 0x7f322c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.841+0000 7f32437fe640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f322c0a2800 0x7f322c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.841+0000 7f3240ff9640 1 -- 172.21.15.77:0/2513502169 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f323c0f0c60 con 0x7f3244102290 2024-03-16T09:39:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.841+0000 7f32437fe640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f322c0a2800 0x7f322c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3244101730 tx=0x7f3238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:58.845+0000 7f3240ff9640 1 -- 172.21.15.77:0/2513502169 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f323c04b020 con 0x7f3244102290 2024-03-16T09:39:59.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.057+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f324402d1b0 con 0x7f3244102290 2024-03-16T09:39:59.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/343704581' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] up:active 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: Cluster is now healthy 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-16T09:39:59.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:39:59 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2513502169' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T09:39:59.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T09:39:59.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/343704581' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T09:39:59.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] up:active 2024-03-16T09:39:59.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T09:39:59.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T09:39:59.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: Cluster is now healthy 2024-03-16T09:39:59.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-16T09:39:59.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-16T09:39:59.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-16T09:39:59.510 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:39:59 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2513502169' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T09:39:59.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/343704581' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] up:active 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: Cluster is now healthy 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-16T09:39:59.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:39:59 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2513502169' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T09:39:59.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.720+0000 7f3240ff9640 1 -- 172.21.15.77:0/2513502169 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7f323c0bde50 con 0x7f3244102290 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.723+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f322c0a2800 msgr2=0x7f322c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.723+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f322c0a2800 0x7f322c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3244101730 tx=0x7f3238005f90 comp rx=0 tx=0).stop 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.723+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 msgr2=0x7f324419ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.723+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f324419ef40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f323c02c700 tx=0x7f323c0328a0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.723+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 shutdown_connections 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.723+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f322c0a2800 0x7f322c0a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.724+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3244102290 0x7f324419ef40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.724+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32441018d0 0x7f3244197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.724+0000 7f3249bb3640 1 --2- 172.21.15.77:0/2513502169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32441006e0 0x7f3244197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.724+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 >> 172.21.15.77:0/2513502169 conn(0x7f32440fbc70 msgr2=0x7f3244104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.724+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 shutdown_connections 2024-03-16T09:39:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.724+0000 7f3249bb3640 1 -- 172.21.15.77:0/2513502169 wait complete. 2024-03-16T09:39:59.740 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-16T09:39:59.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-16T09:39:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 -- 172.21.15.77:0/3934814979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec100700 msgr2=0x7fdbec100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3934814979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec100700 0x7fdbec100b20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc009900 tx=0x7fdbdc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:39:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 -- 172.21.15.77:0/3934814979 shutdown_connections 2024-03-16T09:39:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3934814979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbec1022b0 0x7fdbec108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3934814979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbec1018f0 0x7fdbec101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3934814979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec100700 0x7fdbec100b20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 -- 172.21.15.77:0/3934814979 >> 172.21.15.77:0/3934814979 conn(0x7fdbec0fbc90 msgr2=0x7fdbec0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:39:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 -- 172.21.15.77:0/3934814979 shutdown_connections 2024-03-16T09:39:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 -- 172.21.15.77:0/3934814979 wait complete. 2024-03-16T09:39:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.847+0000 7fdbf1153640 1 Processor -- start 2024-03-16T09:39:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 -- start start 2024-03-16T09:39:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbec100700 0x7fdbec10b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec1018f0 0x7fdbec1096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbec1022b0 0x7fdbec109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec10d350 con 0x7fdbec1018f0 2024-03-16T09:39:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec10d1d0 con 0x7fdbec100700 2024-03-16T09:39:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec10d4d0 con 0x7fdbec1022b0 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec1018f0 0x7fdbec1096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec1018f0 0x7fdbec1096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59630/0 (socket says 172.21.15.77:59630) 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 -- 172.21.15.77:0/3690068260 learned_addr learned my addr 172.21.15.77:0/3690068260 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbead9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbec100700 0x7fdbec10b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbeb59e640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbec1022b0 0x7fdbec109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 -- 172.21.15.77:0/3690068260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbec1022b0 msgr2=0x7fdbec109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbec1022b0 0x7fdbec109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 -- 172.21.15.77:0/3690068260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbec100700 msgr2=0x7fdbec10b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbec100700 0x7fdbec10b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 -- 172.21.15.77:0/3690068260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbec10a490 con 0x7fdbec1018f0 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbead9d640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbec100700 0x7fdbec10b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbea59c640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec1018f0 0x7fdbec1096c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000e9f0 tx=0x7fdbe000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbd3fff640 1 -- 172.21.15.77:0/3690068260 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe000cd80 con 0x7fdbec1018f0 2024-03-16T09:39:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbd3fff640 1 -- 172.21.15.77:0/3690068260 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbe00054d0 con 0x7fdbec1018f0 2024-03-16T09:39:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.848+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbec1a1bb0 con 0x7fdbec1018f0 2024-03-16T09:39:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.849+0000 7fdbd3fff640 1 -- 172.21.15.77:0/3690068260 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0010490 con 0x7fdbec1018f0 2024-03-16T09:39:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.849+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbec1a20b0 con 0x7fdbec1018f0 2024-03-16T09:39:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.849+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbec107b10 con 0x7fdbec1018f0 2024-03-16T09:39:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.850+0000 7fdbd3fff640 1 -- 172.21.15.77:0/3690068260 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdbe0005020 con 0x7fdbec1018f0 2024-03-16T09:39:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.851+0000 7fdbd3fff640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdbd40a2800 0x7fdbd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:39:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.851+0000 7fdbead9d640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdbd40a2800 0x7fdbd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:39:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.851+0000 7fdbd3fff640 1 -- 172.21.15.77:0/3690068260 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdbe00cecc0 con 0x7fdbec1018f0 2024-03-16T09:39:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.851+0000 7fdbead9d640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdbd40a2800 0x7fdbd40a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc005ab0 tx=0x7fdbdc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:39:59.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:39:59.855+0000 7fdbd3fff640 1 -- 172.21.15.77:0/3690068260 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdbe009beb0 con 0x7fdbec1018f0 2024-03-16T09:40:00.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.056+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdbec061c20 con 0x7fdbd40a2800 2024-03-16T09:40:00.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.075+0000 7fdbd3fff640 1 -- 172.21.15.77:0/3690068260 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdbec061c20 con 0x7fdbd40a2800 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdbd40a2800 msgr2=0x7fdbd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdbd40a2800 0x7fdbd40a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc005ab0 tx=0x7fdbdc005fd0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec1018f0 msgr2=0x7fdbec1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec1018f0 0x7fdbec1096c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000e9f0 tx=0x7fdbe000edf0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 shutdown_connections 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdbd40a2800 0x7fdbd40a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbec1022b0 0x7fdbec109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec1018f0 0x7fdbec1096c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 --2- 172.21.15.77:0/3690068260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbec100700 0x7fdbec10b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 >> 172.21.15.77:0/3690068260 conn(0x7fdbec0fbc90 msgr2=0x7fdbec104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:00.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 shutdown_connections 2024-03-16T09:40:00.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.077+0000 7fdbf1153640 1 -- 172.21.15.77:0/3690068260 wait complete. 2024-03-16T09:40:00.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.196+0000 7f41d1005640 1 -- 172.21.15.77:0/574171175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 msgr2=0x7f41cc100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.196+0000 7f41d1005640 1 --2- 172.21.15.77:0/574171175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc100b00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009900 tx=0x7f41bc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- 172.21.15.77:0/574171175 shutdown_connections 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 --2- 172.21.15.77:0/574171175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41cc102290 0x7f41cc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 --2- 172.21.15.77:0/574171175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41cc1018d0 0x7f41cc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 --2- 172.21.15.77:0/574171175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc100b00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- 172.21.15.77:0/574171175 >> 172.21.15.77:0/574171175 conn(0x7f41cc0fbc70 msgr2=0x7f41cc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- 172.21.15.77:0/574171175 shutdown_connections 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- 172.21.15.77:0/574171175 wait complete. 2024-03-16T09:40:00.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 Processor -- start 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- start start 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41cc1018d0 0x7f41cc196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41cc102290 0x7f41cc19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc10c210 con 0x7f41cc1006e0 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc10c090 con 0x7f41cc102290 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41d1005640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc10c390 con 0x7f41cc1018d0 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.197+0000 7f41cad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59652/0 (socket says 172.21.15.77:59652) 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 -- 172.21.15.77:0/2367603508 learned_addr learned my addr 172.21.15.77:0/2367603508 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41c3fff640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41cc1018d0 0x7f41cc196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cb59e640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41cc102290 0x7f41cc19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 -- 172.21.15.77:0/2367603508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41cc1018d0 msgr2=0x7f41cc196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41cc1018d0 0x7f41cc196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 -- 172.21.15.77:0/2367603508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41cc102290 msgr2=0x7f41cc19df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41cc102290 0x7f41cc19df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 -- 172.21.15.77:0/2367603508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc19e5b0 con 0x7f41cc1006e0 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41cad9d640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc196990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f41bc000c00 tx=0x7f41bc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41c8d99640 1 -- 172.21.15.77:0/2367603508 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc043020 con 0x7f41cc1006e0 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41cc19e840 con 0x7f41cc1006e0 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41c8d99640 1 -- 172.21.15.77:0/2367603508 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41bc004350 con 0x7f41cc1006e0 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41c8d99640 1 -- 172.21.15.77:0/2367603508 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc003b80 con 0x7f41cc1006e0 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.198+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41cc19ed90 con 0x7f41cc1006e0 2024-03-16T09:40:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.199+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41cc107b20 con 0x7f41cc1006e0 2024-03-16T09:40:00.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.200+0000 7f41c8d99640 1 -- 172.21.15.77:0/2367603508 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f41bc046020 con 0x7f41cc1006e0 2024-03-16T09:40:00.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.201+0000 7f41c8d99640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f41b80a2800 0x7f41b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.201+0000 7f41c3fff640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f41b80a2800 0x7f41b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.201+0000 7f41c8d99640 1 -- 172.21.15.77:0/2367603508 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f41bc0f0a30 con 0x7f41cc1006e0 2024-03-16T09:40:00.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.201+0000 7f41c3fff640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f41b80a2800 0x7f41b80a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f41cc197ea0 tx=0x7f41b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:00.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.205+0000 7f41c8d99640 1 -- 172.21.15.77:0/2367603508 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41bc04d020 con 0x7f41cc1006e0 2024-03-16T09:40:00.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.409+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f41cc061cb0 con 0x7f41b80a2800 2024-03-16T09:40:00.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.432+0000 7f41c8d99640 1 -- 172.21.15.77:0/2367603508 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f41cc061cb0 con 0x7f41b80a2800 2024-03-16T09:40:00.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f41b80a2800 msgr2=0x7f41b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f41b80a2800 0x7f41b80a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f41cc197ea0 tx=0x7f41b4005f90 comp rx=0 tx=0).stop 2024-03-16T09:40:00.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 msgr2=0x7f41cc196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc196990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f41bc000c00 tx=0x7f41bc005dc0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 shutdown_connections 2024-03-16T09:40:00.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f41b80a2800 0x7f41b80a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41cc102290 0x7f41cc19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41cc1018d0 0x7f41cc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 --2- 172.21.15.77:0/2367603508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41cc1006e0 0x7f41cc196990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 >> 172.21.15.77:0/2367603508 conn(0x7f41cc0fbc70 msgr2=0x7f41cc104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:00.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 shutdown_connections 2024-03-16T09:40:00.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.434+0000 7f41d1005640 1 -- 172.21.15.77:0/2367603508 wait complete. 2024-03-16T09:40:00.447 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:40:00.060+0000 7fe301ab1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T09:40:00.447 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:40:00.060+0000 7fe301ab1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T09:40:00.447 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:40:00.060+0000 7fe301ab1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T09:40:00.447 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:40:00.060+0000 7fe301ab1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T09:40:00.447 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:40:00.060+0000 7fe301ab1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T09:40:00.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.552+0000 7f7bbbc26640 1 -- 172.21.15.77:0/1721649699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41018d0 msgr2=0x7f7bb4101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.552+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/1721649699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41018d0 0x7f7bb4101d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009900 tx=0x7f7ba402bc30 comp rx=0 tx=0).stop 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.552+0000 7f7bbbc26640 1 -- 172.21.15.77:0/1721649699 shutdown_connections 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.552+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/1721649699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb4102290 0x7f7bb4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.552+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/1721649699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41018d0 0x7f7bb4101d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.552+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/1721649699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb41006e0 0x7f7bb4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.552+0000 7f7bbbc26640 1 -- 172.21.15.77:0/1721649699 >> 172.21.15.77:0/1721649699 conn(0x7f7bb40fbc70 msgr2=0x7f7bb40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 -- 172.21.15.77:0/1721649699 shutdown_connections 2024-03-16T09:40:00.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 -- 172.21.15.77:0/1721649699 wait complete. 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 Processor -- start 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 -- start start 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41006e0 0x7f7bb40694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb41018d0 0x7f7bb419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb4102290 0x7f7bb419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb410c210 con 0x7f7bb41006e0 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb410c090 con 0x7f7bb41018d0 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bbbc26640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb410c390 con 0x7f7bb4102290 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bb99c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41006e0 0x7f7bb40694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bb99c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41006e0 0x7f7bb40694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59670/0 (socket says 172.21.15.77:59670) 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bb99c2640 1 -- 172.21.15.77:0/2117945106 learned_addr learned my addr 172.21.15.77:0/2117945106 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bb91c1640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb41018d0 0x7f7bb419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bba1c3640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb4102290 0x7f7bb419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bb99c2640 1 -- 172.21.15.77:0/2117945106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb4102290 msgr2=0x7f7bb419edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.553+0000 7f7bb99c2640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb4102290 0x7f7bb419edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bb99c2640 1 -- 172.21.15.77:0/2117945106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb41018d0 msgr2=0x7f7bb419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bb99c2640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb41018d0 0x7f7bb419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bb99c2640 1 -- 172.21.15.77:0/2117945106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb419f6c0 con 0x7f7bb41006e0 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bb91c1640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb41018d0 0x7f7bb419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bba1c3640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb4102290 0x7f7bb419edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bb99c2640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41006e0 0x7f7bb40694b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d950 tx=0x7f7bb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0017020 con 0x7f7bb41006e0 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb419f950 con 0x7f7bb41006e0 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bb0009c30 con 0x7f7bb41006e0 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb000f620 con 0x7f7bb41006e0 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.554+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb419fea0 con 0x7f7bb41006e0 2024-03-16T09:40:00.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.555+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb4107b40 con 0x7f7bb41006e0 2024-03-16T09:40:00.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.556+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7bb001a020 con 0x7f7bb41006e0 2024-03-16T09:40:00.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.556+0000 7f7baaffd640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.557+0000 7f7bb91c1640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.557+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7bb00ce930 con 0x7f7bb41006e0 2024-03-16T09:40:00.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.557+0000 7f7bb91c1640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4069220 tx=0x7f7ba402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:00.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.561+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7bb0021020 con 0x7f7bb41006e0 2024-03-16T09:40:00.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.741+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7bb0097a90 con 0x7f7bb41006e0 2024-03-16T09:40:00.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.775+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f7bb4061cb0 con 0x7f7ba00a2800 2024-03-16T09:40:00.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.779+0000 7f7baaffd640 1 -- 172.21.15.77:0/2117945106 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7bb4061cb0 con 0x7f7ba00a2800 2024-03-16T09:40:00.782 INFO:teuthology.orchestra.run.smithi077.stdout:/volumes/qa/sv_0/d6cb7cbe-a826-46f6-811e-8b60ec9c6a86 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ba00a2800 msgr2=0x7f7ba00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4069220 tx=0x7f7ba402c7e0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41006e0 msgr2=0x7f7bb40694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41006e0 0x7f7bb40694b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d950 tx=0x7f7bb000de10 comp rx=0 tx=0).stop 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 shutdown_connections 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb4102290 0x7f7bb419edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb41018d0 0x7f7bb419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 --2- 172.21.15.77:0/2117945106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb41006e0 0x7f7bb40694b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 >> 172.21.15.77:0/2117945106 conn(0x7f7bb40fbc70 msgr2=0x7f7bb4104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:00.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 shutdown_connections 2024-03-16T09:40:00.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.781+0000 7f7bbbc26640 1 -- 172.21.15.77:0/2117945106 wait complete. 2024-03-16T09:40:00.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- 172.21.15.77:0/3309964240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c104ba0 msgr2=0x7fb06c106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3309964240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c104ba0 0x7fb06c106f90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb068009900 tx=0x7fb06802bc30 comp rx=0 tx=0).stop 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- 172.21.15.77:0/3309964240 shutdown_connections 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3309964240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb06c1074d0 0x7fb06c109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3309964240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c104ba0 0x7fb06c106f90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3309964240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb06c102270 0x7fb06c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- 172.21.15.77:0/3309964240 >> 172.21.15.77:0/3309964240 conn(0x7fb06c0fbc80 msgr2=0x7fb06c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:00.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- 172.21.15.77:0/3309964240 shutdown_connections 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- 172.21.15.77:0/3309964240 wait complete. 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 Processor -- start 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- start start 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb06c102270 0x7fb06c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb06c104ba0 0x7fb06c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c1074d0 0x7fb06c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c10d310 con 0x7fb06c1074d0 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.898+0000 7fb0744d3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c10d190 con 0x7fb06c104ba0 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb0744d3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c10d490 con 0x7fb06c102270 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c1074d0 0x7fb06c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c1074d0 0x7fb06c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59692/0 (socket says 172.21.15.77:59692) 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 -- 172.21.15.77:0/3618850232 learned_addr learned my addr 172.21.15.77:0/3618850232 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb071a6e640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb06c104ba0 0x7fb06c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb07226f640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb06c102270 0x7fb06c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 -- 172.21.15.77:0/3618850232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb06c102270 msgr2=0x7fb06c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb06c102270 0x7fb06c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 -- 172.21.15.77:0/3618850232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb06c104ba0 msgr2=0x7fb06c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb06c104ba0 0x7fb06c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 -- 172.21.15.77:0/3618850232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06c19f6b0 con 0x7fb06c1074d0 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb07226f640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb06c102270 0x7fb06c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb072a70640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c1074d0 0x7fb06c19f040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb064009fd0 tx=0x7fb06400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb0637fe640 1 -- 172.21.15.77:0/3618850232 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064017070 con 0x7fb06c1074d0 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb0637fe640 1 -- 172.21.15.77:0/3618850232 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb064003e20 con 0x7fb06c1074d0 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06c19f9a0 con 0x7fb06c1074d0 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb0637fe640 1 -- 172.21.15.77:0/3618850232 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064010460 con 0x7fb06c1074d0 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.899+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06c19fec0 con 0x7fb06c1074d0 2024-03-16T09:40:00.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.900+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb06c0fd770 con 0x7fb06c1074d0 2024-03-16T09:40:00.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.901+0000 7fb0637fe640 1 -- 172.21.15.77:0/3618850232 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb064003970 con 0x7fb06c1074d0 2024-03-16T09:40:00.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.902+0000 7fb0637fe640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0580a2910 0x7fb0580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:00.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.902+0000 7fb07226f640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0580a2910 0x7fb0580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:00.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.902+0000 7fb0637fe640 1 -- 172.21.15.77:0/3618850232 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb0640ce770 con 0x7fb06c1074d0 2024-03-16T09:40:00.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.902+0000 7fb07226f640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0580a2910 0x7fb0580a4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005d00 tx=0x7fb05c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:00.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:00.907+0000 7fb0637fe640 1 -- 172.21.15.77:0/3618850232 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb06409b860 con 0x7fb06c1074d0 2024-03-16T09:40:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2513502169' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T09:40:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] up:active 2024-03-16T09:40:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-mon[32171]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-16T09:40:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-mon[32171]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T09:40:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-mon[32171]: pgmap v287: 97 pgs: 13 creating+peering, 72 active+clean, 12 unknown; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 7 op/s 2024-03-16T09:40:01.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:00 smithi077 ceph-mon[32171]: overall HEALTH_OK 2024-03-16T09:40:01.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:00 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2513502169' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T09:40:01.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:00 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] up:active 2024-03-16T09:40:01.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:00 smithi129 ceph-mon[35841]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-16T09:40:01.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:00 smithi129 ceph-mon[35841]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T09:40:01.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:00 smithi129 ceph-mon[35841]: pgmap v287: 97 pgs: 13 creating+peering, 72 active+clean, 12 unknown; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 7 op/s 2024-03-16T09:40:01.052 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:00 smithi129 ceph-mon[35841]: overall HEALTH_OK 2024-03-16T09:40:01.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.110+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb06c061c20 con 0x7fb0580a2910 2024-03-16T09:40:01.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.131+0000 7fb0637fe640 1 -- 172.21.15.77:0/3618850232 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb06c061c20 con 0x7fb0580a2910 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0580a2910 msgr2=0x7fb0580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0580a2910 0x7fb0580a4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005d00 tx=0x7fb05c005c70 comp rx=0 tx=0).stop 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c1074d0 msgr2=0x7fb06c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c1074d0 0x7fb06c19f040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb064009fd0 tx=0x7fb06400b260 comp rx=0 tx=0).stop 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 shutdown_connections 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0580a2910 0x7fb0580a4dd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb06c1074d0 0x7fb06c19f040 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb06c104ba0 0x7fb06c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 --2- 172.21.15.77:0/3618850232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb06c102270 0x7fb06c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 >> 172.21.15.77:0/3618850232 conn(0x7fb06c0fbc80 msgr2=0x7fb06c102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 shutdown_connections 2024-03-16T09:40:01.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.133+0000 7fb0744d3640 1 -- 172.21.15.77:0/3618850232 wait complete. 2024-03-16T09:40:01.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-16T09:40:01.189 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: 2024-03-16T09:40:00.843+0000 7f66b307e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: 2024-03-16T09:40:00.843+0000 7f66b307e700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: 2024-03-16T09:40:00.843+0000 7f66b307e700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2513502169' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] up:active 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-mon[35632]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-mon[35632]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-mon[35632]: pgmap v287: 97 pgs: 13 creating+peering, 72 active+clean, 12 unknown; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 7 op/s 2024-03-16T09:40:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:00 smithi119 ceph-mon[35632]: overall HEALTH_OK 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 -- 172.21.15.77:0/2749606493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 msgr2=0x7f630c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 --2- 172.21.15.77:0/2749606493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c109c30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6304009ca0 tx=0x7f630402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 -- 172.21.15.77:0/2749606493 shutdown_connections 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 --2- 172.21.15.77:0/2749606493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c109c30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 --2- 172.21.15.77:0/2749606493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630c1029e0 0x7f630c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 --2- 172.21.15.77:0/2749606493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f630c1017f0 0x7f630c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 -- 172.21.15.77:0/2749606493 >> 172.21.15.77:0/2749606493 conn(0x7f630c0fcd80 msgr2=0x7f630c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:01.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 -- 172.21.15.77:0/2749606493 shutdown_connections 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 -- 172.21.15.77:0/2749606493 wait complete. 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 Processor -- start 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.250+0000 7f631428a640 1 -- start start 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630c1017f0 0x7f630c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f630c1029e0 0x7f630c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f630c10d320 con 0x7f630c1033a0 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f630c10d1a0 con 0x7f630c1029e0 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f630c10d4a0 con 0x7f630c1017f0 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59708/0 (socket says 172.21.15.77:59708) 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 -- 172.21.15.77:0/3262432344 learned_addr learned my addr 172.21.15.77:0/3262432344 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6311825640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f630c1029e0 0x7f630c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312026640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630c1017f0 0x7f630c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 -- 172.21.15.77:0/3262432344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630c1017f0 msgr2=0x7f630c075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630c1017f0 0x7f630c075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 -- 172.21.15.77:0/3262432344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f630c1029e0 msgr2=0x7f630c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f630c1029e0 0x7f630c073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 -- 172.21.15.77:0/3262432344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630c0749a0 con 0x7f630c1033a0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f6312827640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c074110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f630402c700 tx=0x7f63040328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f63037fe640 1 -- 172.21.15.77:0/3262432344 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304041020 con 0x7f630c1033a0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f63037fe640 1 -- 172.21.15.77:0/3262432344 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6304003e20 con 0x7f630c1033a0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f63037fe640 1 -- 172.21.15.77:0/3262432344 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304039040 con 0x7f630c1033a0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630c1a1a90 con 0x7f630c1033a0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.251+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f630c1a2030 con 0x7f630c1033a0 2024-03-16T09:40:01.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.252+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f630c108b10 con 0x7f630c1033a0 2024-03-16T09:40:01.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.253+0000 7f63037fe640 1 -- 172.21.15.77:0/3262432344 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6304044020 con 0x7f630c1033a0 2024-03-16T09:40:01.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.254+0000 7f63037fe640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f62f80a2910 0x7f62f80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.254+0000 7f6312026640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f62f80a2910 0x7f62f80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.254+0000 7f63037fe640 1 -- 172.21.15.77:0/3262432344 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f63040f0a50 con 0x7f630c1033a0 2024-03-16T09:40:01.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.255+0000 7f6312026640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f62f80a2910 0x7f62f80a4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f630c102840 tx=0x7f6308005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:01.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.259+0000 7f63037fe640 1 -- 172.21.15.77:0/3262432344 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f630404b020 con 0x7f630c1033a0 2024-03-16T09:40:01.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.461+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f630c061cb0 con 0x7f62f80a2910 2024-03-16T09:40:01.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.464+0000 7f63037fe640 1 -- 172.21.15.77:0/3262432344 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f630c061cb0 con 0x7f62f80a2910 2024-03-16T09:40:01.467 INFO:teuthology.orchestra.run.smithi077.stdout:/volumes/qa/sv_1/6f6a8245-b540-4e2c-a675-3ca1aa66fffe 2024-03-16T09:40:01.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f62f80a2910 msgr2=0x7f62f80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f62f80a2910 0x7f62f80a4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f630c102840 tx=0x7f6308005f90 comp rx=0 tx=0).stop 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 msgr2=0x7f630c074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c074110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f630402c700 tx=0x7f63040328a0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 shutdown_connections 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f62f80a2910 0x7f62f80a4dd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f630c1033a0 0x7f630c074110 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f630c1029e0 0x7f630c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 --2- 172.21.15.77:0/3262432344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630c1017f0 0x7f630c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 >> 172.21.15.77:0/3262432344 conn(0x7f630c0fcd80 msgr2=0x7f630c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 shutdown_connections 2024-03-16T09:40:01.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.466+0000 7f631428a640 1 -- 172.21.15.77:0/3262432344 wait complete. 2024-03-16T09:40:01.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 -- 172.21.15.77:0/648984906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a0104bb0 msgr2=0x7ff5a0106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/648984906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a0104bb0 0x7ff5a0106fa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff588009900 tx=0x7ff58802bc30 comp rx=0 tx=0).stop 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 -- 172.21.15.77:0/648984906 shutdown_connections 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/648984906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a01074e0 0x7ff5a0109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/648984906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a0104bb0 0x7ff5a0106fa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/648984906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5a0102280 0x7ff5a0104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 -- 172.21.15.77:0/648984906 >> 172.21.15.77:0/648984906 conn(0x7ff5a00fbc70 msgr2=0x7ff5a00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 -- 172.21.15.77:0/648984906 shutdown_connections 2024-03-16T09:40:01.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.578+0000 7ff5a5a12640 1 -- 172.21.15.77:0/648984906 wait complete. 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 Processor -- start 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 -- start start 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0102280 0x7ff5a0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5a0104bb0 0x7ff5a0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a01074e0 0x7ff5a019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a010d320 con 0x7ff5a01074e0 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a010d1a0 con 0x7ff5a0102280 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff5a5a12640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a010d4a0 con 0x7ff5a0104bb0 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a01074e0 0x7ff5a019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a01074e0 0x7ff5a019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59728/0 (socket says 172.21.15.77:59728) 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 -- 172.21.15.77:0/948737091 learned_addr learned my addr 172.21.15.77:0/948737091 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59effd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0102280 0x7ff5a0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59e7fc640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5a0104bb0 0x7ff5a0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 -- 172.21.15.77:0/948737091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5a0104bb0 msgr2=0x7ff5a0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5a0104bb0 0x7ff5a0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 -- 172.21.15.77:0/948737091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0102280 msgr2=0x7ff5a0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0102280 0x7ff5a0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59f7fe640 1 -- 172.21.15.77:0/948737091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a019f6d0 con 0x7ff5a01074e0 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.579+0000 7ff59effd640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0102280 0x7ff5a0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.580+0000 7ff59f7fe640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a01074e0 0x7ff5a019f060 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff59800e970 tx=0x7ff59800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.580+0000 7ff5a4a10640 1 -- 172.21.15.77:0/948737091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59800ccc0 con 0x7ff5a01074e0 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.580+0000 7ff5a4a10640 1 -- 172.21.15.77:0/948737091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff59800ce60 con 0x7ff5a01074e0 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.580+0000 7ff5a4a10640 1 -- 172.21.15.77:0/948737091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598010430 con 0x7ff5a01074e0 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.580+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a019f9c0 con 0x7ff5a01074e0 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.580+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a019fe90 con 0x7ff5a01074e0 2024-03-16T09:40:01.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.581+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a00fd760 con 0x7ff5a01074e0 2024-03-16T09:40:01.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.582+0000 7ff5a4a10640 1 -- 172.21.15.77:0/948737091 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff598004ff0 con 0x7ff5a01074e0 2024-03-16T09:40:01.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.582+0000 7ff5a4a10640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff58c0a2890 0x7ff58c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.582+0000 7ff59effd640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff58c0a2890 0x7ff58c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.582+0000 7ff5a4a10640 1 -- 172.21.15.77:0/948737091 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff5980cedd0 con 0x7ff5a01074e0 2024-03-16T09:40:01.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.583+0000 7ff59effd640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff58c0a2890 0x7ff58c0a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff590005fd0 tx=0x7ff590009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:01.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.587+0000 7ff5a4a10640 1 -- 172.21.15.77:0/948737091 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff59809bec0 con 0x7ff5a01074e0 2024-03-16T09:40:01.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.801+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5a002d1b0 con 0x7ff5a01074e0 2024-03-16T09:40:01.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.802+0000 7ff5a4a10640 1 -- 172.21.15.77:0/948737091 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11878 (secure 0 0 0) 0x7ff59809b7c0 con 0x7ff5a01074e0 2024-03-16T09:40:01.805 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:40:01.805 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:39:59.725431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24545,24503]},"id":1}]} 2024-03-16T09:40:01.805 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-03-16T09:40:01.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff58c0a2890 msgr2=0x7ff58c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff58c0a2890 0x7ff58c0a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff590005fd0 tx=0x7ff590009760 comp rx=0 tx=0).stop 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a01074e0 msgr2=0x7ff5a019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a01074e0 0x7ff5a019f060 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff59800e970 tx=0x7ff59800ed70 comp rx=0 tx=0).stop 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 shutdown_connections 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff58c0a2890 0x7ff58c0a4d50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a01074e0 0x7ff5a019f060 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5a0104bb0 0x7ff5a0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 --2- 172.21.15.77:0/948737091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0102280 0x7ff5a0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 >> 172.21.15.77:0/948737091 conn(0x7ff5a00fbc70 msgr2=0x7ff5a0104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 shutdown_connections 2024-03-16T09:40:01.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.804+0000 7ff5a5a12640 1 -- 172.21.15.77:0/948737091 wait complete. 2024-03-16T09:40:01.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:40:01.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 -- 172.21.15.77:0/1966899911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc101910 msgr2=0x7fd9dc101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/1966899911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc101910 0x7fd9dc101d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009900 tx=0x7fd9cc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:40:01.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 -- 172.21.15.77:0/1966899911 shutdown_connections 2024-03-16T09:40:01.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/1966899911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9dc1022d0 0x7fd9dc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/1966899911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc101910 0x7fd9dc101d90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/1966899911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc100720 0x7fd9dc100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 -- 172.21.15.77:0/1966899911 >> 172.21.15.77:0/1966899911 conn(0x7fd9dc0fbc90 msgr2=0x7fd9dc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 -- 172.21.15.77:0/1966899911 shutdown_connections 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 -- 172.21.15.77:0/1966899911 wait complete. 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 Processor -- start 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.918+0000 7fd9e1b10640 1 -- start start 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc100720 0x7fd9dc06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9dc101910 0x7fd9dc19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc1022d0 0x7fd9dc19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd9dc10c250 con 0x7fd9dc1022d0 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd9dc10c0d0 con 0x7fd9dc100720 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9dc10c3d0 con 0x7fd9dc101910 2024-03-16T09:40:01.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc1022d0 0x7fd9dc19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc1022d0 0x7fd9dc19edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59744/0 (socket says 172.21.15.77:59744) 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 -- 172.21.15.77:0/999281129 learned_addr learned my addr 172.21.15.77:0/999281129 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9db7fe640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc100720 0x7fd9dc06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9daffd640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9dc101910 0x7fd9dc19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 -- 172.21.15.77:0/999281129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9dc101910 msgr2=0x7fd9dc19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9dc101910 0x7fd9dc19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 -- 172.21.15.77:0/999281129 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc100720 msgr2=0x7fd9dc06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc100720 0x7fd9dc06a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 -- 172.21.15.77:0/999281129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9dc19f680 con 0x7fd9dc1022d0 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9dbfff640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc1022d0 0x7fd9dc19edc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400eb10 tx=0x7fd9d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9d8ff9640 1 -- 172.21.15.77:0/999281129 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4004b20 con 0x7fd9dc1022d0 2024-03-16T09:40:01.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9dc19f970 con 0x7fd9dc1022d0 2024-03-16T09:40:01.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9d8ff9640 1 -- 172.21.15.77:0/999281129 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9d4004cc0 con 0x7fd9dc1022d0 2024-03-16T09:40:01.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9d8ff9640 1 -- 172.21.15.77:0/999281129 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4010690 con 0x7fd9dc1022d0 2024-03-16T09:40:01.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.919+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9dc19fe90 con 0x7fd9dc1022d0 2024-03-16T09:40:01.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.920+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9dc107ae0 con 0x7fd9dc1022d0 2024-03-16T09:40:01.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.921+0000 7fd9d8ff9640 1 -- 172.21.15.77:0/999281129 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd9d4005020 con 0x7fd9dc1022d0 2024-03-16T09:40:01.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.922+0000 7fd9d8ff9640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9c80a2890 0x7fd9c80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:01.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.922+0000 7fd9db7fe640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9c80a2890 0x7fd9c80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:01.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.922+0000 7fd9d8ff9640 1 -- 172.21.15.77:0/999281129 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd9d40cf100 con 0x7fd9dc1022d0 2024-03-16T09:40:01.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.922+0000 7fd9db7fe640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9c80a2890 0x7fd9c80a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc101770 tx=0x7fd9c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:01.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:01.926+0000 7fd9d8ff9640 1 -- 172.21.15.77:0/999281129 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9d409c1f0 con 0x7fd9dc1022d0 2024-03-16T09:40:02.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:01 smithi077 ceph-mon[32171]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:01 smithi077 ceph-mon[32171]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:01 smithi077 ceph-mon[32171]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T09:40:02.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:01 smithi077 ceph-mon[32171]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:01 smithi129 ceph-mon[35841]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:01 smithi129 ceph-mon[35841]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:01 smithi129 ceph-mon[35841]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T09:40:02.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:01 smithi129 ceph-mon[35841]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.105+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9dc02d3c0 con 0x7fd9dc1022d0 2024-03-16T09:40:02.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.107+0000 7fd9d8ff9640 1 -- 172.21.15.77:0/999281129 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22231 (secure 0 0 0) 0x7fd9d409baf0 con 0x7fd9dc1022d0 2024-03-16T09:40:02.111 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:40:02.111 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":86,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:57.172185+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:40:02.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9c80a2890 msgr2=0x7fd9c80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9c80a2890 0x7fd9c80a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc101770 tx=0x7fd9c4005f90 comp rx=0 tx=0).stop 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc1022d0 msgr2=0x7fd9dc19edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc1022d0 0x7fd9dc19edc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400eb10 tx=0x7fd9d400c2d0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 shutdown_connections 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9c80a2890 0x7fd9c80a4d50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9dc1022d0 0x7fd9dc19edc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9dc101910 0x7fd9dc19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 --2- 172.21.15.77:0/999281129 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc100720 0x7fd9dc06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 >> 172.21.15.77:0/999281129 conn(0x7fd9dc0fbc90 msgr2=0x7fd9dc104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 shutdown_connections 2024-03-16T09:40:02.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.109+0000 7fd9e1b10640 1 -- 172.21.15.77:0/999281129 wait complete. 2024-03-16T09:40:02.123 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T09:40:02.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T09:40:02.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:01 smithi119 ceph-mon[35632]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:01 smithi119 ceph-mon[35632]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:01 smithi119 ceph-mon[35632]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T09:40:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:01 smithi119 ceph-mon[35632]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:02.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- 172.21.15.77:0/3741064671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8101350 msgr2=0x7fd9e8108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3741064671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8101350 0x7fd9e8108bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009930 tx=0x7fd9e002bc30 comp rx=0 tx=0).stop 2024-03-16T09:40:02.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- 172.21.15.77:0/3741064671 shutdown_connections 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3741064671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8101350 0x7fd9e8108bf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3741064671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9e8100990 0x7fd9e8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3741064671 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9e8101cf0 0x7fd9e8102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- 172.21.15.77:0/3741064671 >> 172.21.15.77:0/3741064671 conn(0x7fd9e80fb400 msgr2=0x7fd9e80fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- 172.21.15.77:0/3741064671 shutdown_connections 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- 172.21.15.77:0/3741064671 wait complete. 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 Processor -- start 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- start start 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8100990 0x7fd9e81003f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9e8101350 0x7fd9e819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9e8101cf0 0x7fd9e819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e810c250 con 0x7fd9e8100990 2024-03-16T09:40:02.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e810c0d0 con 0x7fd9e8101cf0 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.223+0000 7fd9efe61640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e810c3d0 con 0x7fd9e8101350 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8100990 0x7fd9e81003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8100990 0x7fd9e81003f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59770/0 (socket says 172.21.15.77:59770) 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 -- 172.21.15.77:0/3756660035 learned_addr learned my addr 172.21.15.77:0/3756660035 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9ed3fc640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9e8101350 0x7fd9e819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9ee3fe640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9e8101cf0 0x7fd9e819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 -- 172.21.15.77:0/3756660035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9e8101350 msgr2=0x7fd9e819c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9e8101350 0x7fd9e819c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 -- 172.21.15.77:0/3756660035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9e8101cf0 msgr2=0x7fd9e819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9e8101cf0 0x7fd9e819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 -- 172.21.15.77:0/3756660035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e819f680 con 0x7fd9e8100990 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9edbfd640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8100990 0x7fd9e81003f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400eb70 tx=0x7fd9e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9deffd640 1 -- 172.21.15.77:0/3756660035 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4003b70 con 0x7fd9e8100990 2024-03-16T09:40:02.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e819f910 con 0x7fd9e8100990 2024-03-16T09:40:02.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9deffd640 1 -- 172.21.15.77:0/3756660035 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9e400ee70 con 0x7fd9e8100990 2024-03-16T09:40:02.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9deffd640 1 -- 172.21.15.77:0/3756660035 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4010650 con 0x7fd9e8100990 2024-03-16T09:40:02.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.224+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e819fe10 con 0x7fd9e8100990 2024-03-16T09:40:02.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.225+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e8068a20 con 0x7fd9e8100990 2024-03-16T09:40:02.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.226+0000 7fd9deffd640 1 -- 172.21.15.77:0/3756660035 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd9e4004020 con 0x7fd9e8100990 2024-03-16T09:40:02.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.227+0000 7fd9deffd640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9d40a2890 0x7fd9d40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.227+0000 7fd9ed3fc640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9d40a2890 0x7fd9d40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.227+0000 7fd9deffd640 1 -- 172.21.15.77:0/3756660035 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd9e40cf0b0 con 0x7fd9e8100990 2024-03-16T09:40:02.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.227+0000 7fd9ed3fc640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9d40a2890 0x7fd9d40a4d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8100160 tx=0x7fd9d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:02.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.231+0000 7fd9deffd640 1 -- 172.21.15.77:0/3756660035 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9e409c1a0 con 0x7fd9e8100990 2024-03-16T09:40:02.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.406+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9e8061ce0 con 0x7fd9d40a2890 2024-03-16T09:40:02.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.414+0000 7fd9deffd640 1 -- 172.21.15.77:0/3756660035 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258768 (secure 0 0 0) 0x7fd9e8061ce0 con 0x7fd9d40a2890 2024-03-16T09:40:02.420 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:40:02.422 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9d40a2890 msgr2=0x7fd9d40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9d40a2890 0x7fd9d40a4d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8100160 tx=0x7fd9d8005f90 comp rx=0 tx=0).stop 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8100990 msgr2=0x7fd9e81003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8100990 0x7fd9e81003f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400eb70 tx=0x7fd9e400c6a0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 shutdown_connections 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd9d40a2890 0x7fd9d40a4d50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9e8101cf0 0x7fd9e819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9e8101350 0x7fd9e819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 --2- 172.21.15.77:0/3756660035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e8100990 0x7fd9e81003f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 >> 172.21.15.77:0/3756660035 conn(0x7fd9e80fb400 msgr2=0x7fd9e8104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:02.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 shutdown_connections 2024-03-16T09:40:02.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.417+0000 7fd9efe61640 1 -- 172.21.15.77:0/3756660035 wait complete. 2024-03-16T09:40:02.431 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-16T09:40:01.771713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596909,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":139,"num_read_kb":121,"num_write":306,"num_write_kb":2979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":238,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357816,"kb_used_data":12764,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584008,"statfs":{"total":1151940427776,"available":1151574024192,"internally_reserved":0,"allocated":13070336,"data_stored":7082672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6541,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":125,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.608347"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269281+0000","last_change":"2024-03-16T09:39:56.184389+0000","last_active":"2024-03-16T09:39:57.269281+0000","last_peered":"2024-03-16T09:39:57.269281+0000","last_clean":"2024-03-16T09:39:57.269281+0000","last_became_active":"2024-03-16T09:39:56.184129+0000","last_became_peered":"2024-03-16T09:39:56.184129+0000","last_unstale":"2024-03-16T09:39:57.269281+0000","last_undegraded":"2024-03-16T09:39:57.269281+0000","last_fullsized":"2024-03-16T09:39:57.269281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:33.786788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268573+0000","last_change":"2024-03-16T09:39:56.193451+0000","last_active":"2024-03-16T09:39:57.268573+0000","last_peered":"2024-03-16T09:39:57.268573+0000","last_clean":"2024-03-16T09:39:57.268573+0000","last_became_active":"2024-03-16T09:39:56.187381+0000","last_became_peered":"2024-03-16T09:39:56.187381+0000","last_unstale":"2024-03-16T09:39:57.268573+0000","last_undegraded":"2024-03-16T09:39:57.268573+0000","last_fullsized":"2024-03-16T09:39:57.268573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:51.915654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268387+0000","last_change":"2024-03-16T09:39:56.183785+0000","last_active":"2024-03-16T09:39:57.268387+0000","last_peered":"2024-03-16T09:39:57.268387+0000","last_clean":"2024-03-16T09:39:57.268387+0000","last_became_active":"2024-03-16T09:39:56.183421+0000","last_became_peered":"2024-03-16T09:39:56.183421+0000","last_unstale":"2024-03-16T09:39:57.268387+0000","last_undegraded":"2024-03-16T09:39:57.268387+0000","last_fullsized":"2024-03-16T09:39:57.268387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:47.259139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268934+0000","last_change":"2024-03-16T09:39:56.195743+0000","last_active":"2024-03-16T09:39:57.268934+0000","last_peered":"2024-03-16T09:39:57.268934+0000","last_clean":"2024-03-16T09:39:57.268934+0000","last_became_active":"2024-03-16T09:39:56.194818+0000","last_became_peered":"2024-03-16T09:39:56.194818+0000","last_unstale":"2024-03-16T09:39:57.268934+0000","last_undegraded":"2024-03-16T09:39:57.268934+0000","last_fullsized":"2024-03-16T09:39:57.268934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:39:43.123879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191373+0000","last_change":"2024-03-16T09:39:56.184970+0000","last_active":"2024-03-16T09:39:57.191373+0000","last_peered":"2024-03-16T09:39:57.191373+0000","last_clean":"2024-03-16T09:39:57.191373+0000","last_became_active":"2024-03-16T09:39:56.184632+0000","last_became_peered":"2024-03-16T09:39:56.184632+0000","last_unstale":"2024-03-16T09:39:57.191373+0000","last_undegraded":"2024-03-16T09:39:57.191373+0000","last_fullsized":"2024-03-16T09:39:57.191373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:46:51.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269044+0000","last_change":"2024-03-16T09:39:56.199688+0000","last_active":"2024-03-16T09:39:57.269044+0000","last_peered":"2024-03-16T09:39:57.269044+0000","last_clean":"2024-03-16T09:39:57.269044+0000","last_became_active":"2024-03-16T09:39:56.199094+0000","last_became_peered":"2024-03-16T09:39:56.199094+0000","last_unstale":"2024-03-16T09:39:57.269044+0000","last_undegraded":"2024-03-16T09:39:57.269044+0000","last_fullsized":"2024-03-16T09:39:57.269044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:10.855504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269892+0000","last_change":"2024-03-16T09:39:56.180816+0000","last_active":"2024-03-16T09:39:57.269892+0000","last_peered":"2024-03-16T09:39:57.269892+0000","last_clean":"2024-03-16T09:39:57.269892+0000","last_became_active":"2024-03-16T09:39:56.180499+0000","last_became_peered":"2024-03-16T09:39:56.180499+0000","last_unstale":"2024-03-16T09:39:57.269892+0000","last_undegraded":"2024-03-16T09:39:57.269892+0000","last_fullsized":"2024-03-16T09:39:57.269892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:32:09.444494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185191+0000","last_change":"2024-03-16T09:39:56.189960+0000","last_active":"2024-03-16T09:39:57.185191+0000","last_peered":"2024-03-16T09:39:57.185191+0000","last_clean":"2024-03-16T09:39:57.185191+0000","last_became_active":"2024-03-16T09:39:56.189609+0000","last_became_peered":"2024-03-16T09:39:56.189609+0000","last_unstale":"2024-03-16T09:39:57.185191+0000","last_undegraded":"2024-03-16T09:39:57.185191+0000","last_fullsized":"2024-03-16T09:39:57.185191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:50.219703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182394+0000","last_change":"2024-03-16T09:39:56.188383+0000","last_active":"2024-03-16T09:39:57.182394+0000","last_peered":"2024-03-16T09:39:57.182394+0000","last_clean":"2024-03-16T09:39:57.182394+0000","last_became_active":"2024-03-16T09:39:56.187926+0000","last_became_peered":"2024-03-16T09:39:56.187926+0000","last_unstale":"2024-03-16T09:39:57.182394+0000","last_undegraded":"2024-03-16T09:39:57.182394+0000","last_fullsized":"2024-03-16T09:39:57.182394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:06.422573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269391+0000","last_change":"2024-03-16T09:39:56.201584+0000","last_active":"2024-03-16T09:39:57.269391+0000","last_peered":"2024-03-16T09:39:57.269391+0000","last_clean":"2024-03-16T09:39:57.269391+0000","last_became_active":"2024-03-16T09:39:56.201320+0000","last_became_peered":"2024-03-16T09:39:56.201320+0000","last_unstale":"2024-03-16T09:39:57.269391+0000","last_undegraded":"2024-03-16T09:39:57.269391+0000","last_fullsized":"2024-03-16T09:39:57.269391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:18:00.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269927+0000","last_change":"2024-03-16T09:39:56.191125+0000","last_active":"2024-03-16T09:39:57.269927+0000","last_peered":"2024-03-16T09:39:57.269927+0000","last_clean":"2024-03-16T09:39:57.269927+0000","last_became_active":"2024-03-16T09:39:56.190795+0000","last_became_peered":"2024-03-16T09:39:56.190795+0000","last_unstale":"2024-03-16T09:39:57.269927+0000","last_undegraded":"2024-03-16T09:39:57.269927+0000","last_fullsized":"2024-03-16T09:39:57.269927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:44:10.894995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268723+0000","last_change":"2024-03-16T09:39:56.195652+0000","last_active":"2024-03-16T09:39:57.268723+0000","last_peered":"2024-03-16T09:39:57.268723+0000","last_clean":"2024-03-16T09:39:57.268723+0000","last_became_active":"2024-03-16T09:39:56.195270+0000","last_became_peered":"2024-03-16T09:39:56.195270+0000","last_unstale":"2024-03-16T09:39:57.268723+0000","last_undegraded":"2024-03-16T09:39:57.268723+0000","last_fullsized":"2024-03-16T09:39:57.268723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:03:41.546991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268370+0000","last_change":"2024-03-16T09:39:56.184622+0000","last_active":"2024-03-16T09:39:57.268370+0000","last_peered":"2024-03-16T09:39:57.268370+0000","last_clean":"2024-03-16T09:39:57.268370+0000","last_became_active":"2024-03-16T09:39:56.183532+0000","last_became_peered":"2024-03-16T09:39:56.183532+0000","last_unstale":"2024-03-16T09:39:57.268370+0000","last_undegraded":"2024-03-16T09:39:57.268370+0000","last_fullsized":"2024-03-16T09:39:57.268370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:11:24.574088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268685+0000","last_change":"2024-03-16T09:39:56.184393+0000","last_active":"2024-03-16T09:39:57.268685+0000","last_peered":"2024-03-16T09:39:57.268685+0000","last_clean":"2024-03-16T09:39:57.268685+0000","last_became_active":"2024-03-16T09:39:56.183783+0000","last_became_peered":"2024-03-16T09:39:56.183783+0000","last_unstale":"2024-03-16T09:39:57.268685+0000","last_undegraded":"2024-03-16T09:39:57.268685+0000","last_fullsized":"2024-03-16T09:39:57.268685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:16:22.131116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268036+0000","last_change":"2024-03-16T09:39:56.197365+0000","last_active":"2024-03-16T09:39:57.268036+0000","last_peered":"2024-03-16T09:39:57.268036+0000","last_clean":"2024-03-16T09:39:57.268036+0000","last_became_active":"2024-03-16T09:39:56.197020+0000","last_became_peered":"2024-03-16T09:39:56.197020+0000","last_unstale":"2024-03-16T09:39:57.268036+0000","last_undegraded":"2024-03-16T09:39:57.268036+0000","last_fullsized":"2024-03-16T09:39:57.268036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:29.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.181798+0000","last_change":"2024-03-16T09:39:56.185513+0000","last_active":"2024-03-16T09:39:57.181798+0000","last_peered":"2024-03-16T09:39:57.181798+0000","last_clean":"2024-03-16T09:39:57.181798+0000","last_became_active":"2024-03-16T09:39:56.184384+0000","last_became_peered":"2024-03-16T09:39:56.184384+0000","last_unstale":"2024-03-16T09:39:57.181798+0000","last_undegraded":"2024-03-16T09:39:57.181798+0000","last_fullsized":"2024-03-16T09:39:57.181798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:29.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269258+0000","last_change":"2024-03-16T09:39:56.184536+0000","last_active":"2024-03-16T09:39:57.269258+0000","last_peered":"2024-03-16T09:39:57.269258+0000","last_clean":"2024-03-16T09:39:57.269258+0000","last_became_active":"2024-03-16T09:39:56.184208+0000","last_became_peered":"2024-03-16T09:39:56.184208+0000","last_unstale":"2024-03-16T09:39:57.269258+0000","last_undegraded":"2024-03-16T09:39:57.269258+0000","last_fullsized":"2024-03-16T09:39:57.269258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:14.025625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.189057+0000","last_change":"2024-03-16T09:39:56.184331+0000","last_active":"2024-03-16T09:39:57.189057+0000","last_peered":"2024-03-16T09:39:57.189057+0000","last_clean":"2024-03-16T09:39:57.189057+0000","last_became_active":"2024-03-16T09:39:56.183919+0000","last_became_peered":"2024-03-16T09:39:56.183919+0000","last_unstale":"2024-03-16T09:39:57.189057+0000","last_undegraded":"2024-03-16T09:39:57.189057+0000","last_fullsized":"2024-03-16T09:39:57.189057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:59.692854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268567+0000","last_change":"2024-03-16T09:39:56.185607+0000","last_active":"2024-03-16T09:39:57.268567+0000","last_peered":"2024-03-16T09:39:57.268567+0000","last_clean":"2024-03-16T09:39:57.268567+0000","last_became_active":"2024-03-16T09:39:56.185278+0000","last_became_peered":"2024-03-16T09:39:56.185278+0000","last_unstale":"2024-03-16T09:39:57.268567+0000","last_undegraded":"2024-03-16T09:39:57.268567+0000","last_fullsized":"2024-03-16T09:39:57.268567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:20.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732837+0000","last_change":"2024-03-16T09:39:55.180852+0000","last_active":"2024-03-16T09:39:58.732837+0000","last_peered":"2024-03-16T09:39:58.732837+0000","last_clean":"2024-03-16T09:39:58.732837+0000","last_became_active":"2024-03-16T09:39:55.180496+0000","last_became_peered":"2024-03-16T09:39:55.180496+0000","last_unstale":"2024-03-16T09:39:58.732837+0000","last_undegraded":"2024-03-16T09:39:58.732837+0000","last_fullsized":"2024-03-16T09:39:58.732837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:15:28.548458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185400+0000","last_change":"2024-03-16T09:39:56.190110+0000","last_active":"2024-03-16T09:39:57.185400+0000","last_peered":"2024-03-16T09:39:57.185400+0000","last_clean":"2024-03-16T09:39:57.185400+0000","last_became_active":"2024-03-16T09:39:56.189711+0000","last_became_peered":"2024-03-16T09:39:56.189711+0000","last_unstale":"2024-03-16T09:39:57.185400+0000","last_undegraded":"2024-03-16T09:39:57.185400+0000","last_fullsized":"2024-03-16T09:39:57.185400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:17.353457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185703+0000","last_change":"2024-03-16T09:39:55.171142+0000","last_active":"2024-03-16T09:39:57.185703+0000","last_peered":"2024-03-16T09:39:57.185703+0000","last_clean":"2024-03-16T09:39:57.185703+0000","last_became_active":"2024-03-16T09:39:55.170793+0000","last_became_peered":"2024-03-16T09:39:55.170793+0000","last_unstale":"2024-03-16T09:39:57.185703+0000","last_undegraded":"2024-03-16T09:39:57.185703+0000","last_fullsized":"2024-03-16T09:39:57.185703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:36.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268798+0000","last_change":"2024-03-16T09:39:56.186802+0000","last_active":"2024-03-16T09:39:57.268798+0000","last_peered":"2024-03-16T09:39:57.268798+0000","last_clean":"2024-03-16T09:39:57.268798+0000","last_became_active":"2024-03-16T09:39:56.186398+0000","last_became_peered":"2024-03-16T09:39:56.186398+0000","last_unstale":"2024-03-16T09:39:57.268798+0000","last_undegraded":"2024-03-16T09:39:57.268798+0000","last_fullsized":"2024-03-16T09:39:57.268798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:05:56.209864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731812+0000","last_change":"2024-03-16T09:39:55.181894+0000","last_active":"2024-03-16T09:39:58.731812+0000","last_peered":"2024-03-16T09:39:58.731812+0000","last_clean":"2024-03-16T09:39:58.731812+0000","last_became_active":"2024-03-16T09:39:55.179434+0000","last_became_peered":"2024-03-16T09:39:55.179434+0000","last_unstale":"2024-03-16T09:39:58.731812+0000","last_undegraded":"2024-03-16T09:39:58.731812+0000","last_fullsized":"2024-03-16T09:39:58.731812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:16.153928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182169+0000","last_change":"2024-03-16T09:39:56.188464+0000","last_active":"2024-03-16T09:39:57.182169+0000","last_peered":"2024-03-16T09:39:57.182169+0000","last_clean":"2024-03-16T09:39:57.182169+0000","last_became_active":"2024-03-16T09:39:56.188125+0000","last_became_peered":"2024-03-16T09:39:56.188125+0000","last_unstale":"2024-03-16T09:39:57.182169+0000","last_undegraded":"2024-03-16T09:39:57.182169+0000","last_fullsized":"2024-03-16T09:39:57.182169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:46:49.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191478+0000","last_change":"2024-03-16T09:39:55.179379+0000","last_active":"2024-03-16T09:39:57.191478+0000","last_peered":"2024-03-16T09:39:57.191478+0000","last_clean":"2024-03-16T09:39:57.191478+0000","last_became_active":"2024-03-16T09:39:55.178755+0000","last_became_peered":"2024-03-16T09:39:55.178755+0000","last_unstale":"2024-03-16T09:39:57.191478+0000","last_undegraded":"2024-03-16T09:39:57.191478+0000","last_fullsized":"2024-03-16T09:39:57.191478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:07.131417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.197489+0000","last_change":"2024-03-16T09:39:56.196655+0000","last_active":"2024-03-16T09:39:56.197489+0000","last_peered":"2024-03-16T09:39:56.197489+0000","last_clean":"2024-03-16T09:39:56.197489+0000","last_became_active":"2024-03-16T09:39:56.196344+0000","last_became_peered":"2024-03-16T09:39:56.196344+0000","last_unstale":"2024-03-16T09:39:56.197489+0000","last_undegraded":"2024-03-16T09:39:56.197489+0000","last_fullsized":"2024-03-16T09:39:56.197489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:15.437849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.194714+0000","last_change":"2024-03-16T09:39:55.182012+0000","last_active":"2024-03-16T09:39:56.194714+0000","last_peered":"2024-03-16T09:39:56.194714+0000","last_clean":"2024-03-16T09:39:56.194714+0000","last_became_active":"2024-03-16T09:39:55.178651+0000","last_became_peered":"2024-03-16T09:39:55.178651+0000","last_unstale":"2024-03-16T09:39:56.194714+0000","last_undegraded":"2024-03-16T09:39:56.194714+0000","last_fullsized":"2024-03-16T09:39:56.194714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:50:19.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191077+0000","last_change":"2024-03-16T09:39:56.179645+0000","last_active":"2024-03-16T09:39:57.191077+0000","last_peered":"2024-03-16T09:39:57.191077+0000","last_clean":"2024-03-16T09:39:57.191077+0000","last_became_active":"2024-03-16T09:39:56.179257+0000","last_became_peered":"2024-03-16T09:39:56.179257+0000","last_unstale":"2024-03-16T09:39:57.191077+0000","last_undegraded":"2024-03-16T09:39:57.191077+0000","last_fullsized":"2024-03-16T09:39:57.191077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:30:09.104630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731495+0000","last_change":"2024-03-16T09:39:55.178257+0000","last_active":"2024-03-16T09:39:58.731495+0000","last_peered":"2024-03-16T09:39:58.731495+0000","last_clean":"2024-03-16T09:39:58.731495+0000","last_became_active":"2024-03-16T09:39:55.177984+0000","last_became_peered":"2024-03-16T09:39:55.177984+0000","last_unstale":"2024-03-16T09:39:58.731495+0000","last_undegraded":"2024-03-16T09:39:58.731495+0000","last_fullsized":"2024-03-16T09:39:58.731495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.385624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.198862+0000","last_change":"2024-03-16T09:39:56.196821+0000","last_active":"2024-03-16T09:39:56.198862+0000","last_peered":"2024-03-16T09:39:56.198862+0000","last_clean":"2024-03-16T09:39:56.198862+0000","last_became_active":"2024-03-16T09:39:56.196522+0000","last_became_peered":"2024-03-16T09:39:56.196522+0000","last_unstale":"2024-03-16T09:39:56.198862+0000","last_undegraded":"2024-03-16T09:39:56.198862+0000","last_fullsized":"2024-03-16T09:39:56.198862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:19:42.587126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732000+0000","last_change":"2024-03-16T09:39:55.170290+0000","last_active":"2024-03-16T09:39:58.732000+0000","last_peered":"2024-03-16T09:39:58.732000+0000","last_clean":"2024-03-16T09:39:58.732000+0000","last_became_active":"2024-03-16T09:39:55.169956+0000","last_became_peered":"2024-03-16T09:39:55.169956+0000","last_unstale":"2024-03-16T09:39:58.732000+0000","last_undegraded":"2024-03-16T09:39:58.732000+0000","last_fullsized":"2024-03-16T09:39:58.732000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:01:31.606917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185671+0000","last_change":"2024-03-16T09:39:56.189155+0000","last_active":"2024-03-16T09:39:57.185671+0000","last_peered":"2024-03-16T09:39:57.185671+0000","last_clean":"2024-03-16T09:39:57.185671+0000","last_became_active":"2024-03-16T09:39:56.188827+0000","last_became_peered":"2024-03-16T09:39:56.188827+0000","last_unstale":"2024-03-16T09:39:57.185671+0000","last_undegraded":"2024-03-16T09:39:57.185671+0000","last_fullsized":"2024-03-16T09:39:57.185671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:54.370754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.172364+0000","last_change":"2024-03-16T09:39:55.181886+0000","last_active":"2024-03-16T09:39:56.172364+0000","last_peered":"2024-03-16T09:39:56.172364+0000","last_clean":"2024-03-16T09:39:56.172364+0000","last_became_active":"2024-03-16T09:39:55.179836+0000","last_became_peered":"2024-03-16T09:39:55.179836+0000","last_unstale":"2024-03-16T09:39:56.172364+0000","last_undegraded":"2024-03-16T09:39:56.172364+0000","last_fullsized":"2024-03-16T09:39:56.172364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:52:38.772861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.188909+0000","last_change":"2024-03-16T09:39:56.185892+0000","last_active":"2024-03-16T09:39:57.188909+0000","last_peered":"2024-03-16T09:39:57.188909+0000","last_clean":"2024-03-16T09:39:57.188909+0000","last_became_active":"2024-03-16T09:39:56.185459+0000","last_became_peered":"2024-03-16T09:39:56.185459+0000","last_unstale":"2024-03-16T09:39:57.188909+0000","last_undegraded":"2024-03-16T09:39:57.188909+0000","last_fullsized":"2024-03-16T09:39:57.188909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:10.421054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713808+0000","last_change":"2024-03-16T09:39:55.177607+0000","last_active":"2024-03-16T09:39:57.713808+0000","last_peered":"2024-03-16T09:39:57.713808+0000","last_clean":"2024-03-16T09:39:57.713808+0000","last_became_active":"2024-03-16T09:39:55.177265+0000","last_became_peered":"2024-03-16T09:39:55.177265+0000","last_unstale":"2024-03-16T09:39:57.713808+0000","last_undegraded":"2024-03-16T09:39:57.713808+0000","last_fullsized":"2024-03-16T09:39:57.713808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:47.849041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182281+0000","last_change":"2024-03-16T09:39:56.189172+0000","last_active":"2024-03-16T09:39:57.182281+0000","last_peered":"2024-03-16T09:39:57.182281+0000","last_clean":"2024-03-16T09:39:57.182281+0000","last_became_active":"2024-03-16T09:39:56.188798+0000","last_became_peered":"2024-03-16T09:39:56.188798+0000","last_unstale":"2024-03-16T09:39:57.182281+0000","last_undegraded":"2024-03-16T09:39:57.182281+0000","last_fullsized":"2024-03-16T09:39:57.182281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:02:11.119293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.712618+0000","last_change":"2024-03-16T09:39:55.180369+0000","last_active":"2024-03-16T09:39:57.712618+0000","last_peered":"2024-03-16T09:39:57.712618+0000","last_clean":"2024-03-16T09:39:57.712618+0000","last_became_active":"2024-03-16T09:39:55.180053+0000","last_became_peered":"2024-03-16T09:39:55.180053+0000","last_unstale":"2024-03-16T09:39:57.712618+0000","last_undegraded":"2024-03-16T09:39:57.712618+0000","last_fullsized":"2024-03-16T09:39:57.712618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:44.990028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268679+0000","last_change":"2024-03-16T09:39:56.198454+0000","last_active":"2024-03-16T09:39:57.268679+0000","last_peered":"2024-03-16T09:39:57.268679+0000","last_clean":"2024-03-16T09:39:57.268679+0000","last_became_active":"2024-03-16T09:39:56.198097+0000","last_became_peered":"2024-03-16T09:39:56.198097+0000","last_unstale":"2024-03-16T09:39:57.268679+0000","last_undegraded":"2024-03-16T09:39:57.268679+0000","last_fullsized":"2024-03-16T09:39:57.268679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:12.494363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.195938+0000","last_change":"2024-03-16T09:39:55.174221+0000","last_active":"2024-03-16T09:39:56.195938+0000","last_peered":"2024-03-16T09:39:56.195938+0000","last_clean":"2024-03-16T09:39:56.195938+0000","last_became_active":"2024-03-16T09:39:55.173920+0000","last_became_peered":"2024-03-16T09:39:55.173920+0000","last_unstale":"2024-03-16T09:39:56.195938+0000","last_undegraded":"2024-03-16T09:39:56.195938+0000","last_fullsized":"2024-03-16T09:39:56.195938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:25.108414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182119+0000","last_change":"2024-03-16T09:39:56.196849+0000","last_active":"2024-03-16T09:39:57.182119+0000","last_peered":"2024-03-16T09:39:57.182119+0000","last_clean":"2024-03-16T09:39:57.182119+0000","last_became_active":"2024-03-16T09:39:56.196515+0000","last_became_peered":"2024-03-16T09:39:56.196515+0000","last_unstale":"2024-03-16T09:39:57.182119+0000","last_undegraded":"2024-03-16T09:39:57.182119+0000","last_fullsized":"2024-03-16T09:39:57.182119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:03.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:40:00.065303+0000","last_change":"2024-03-16T09:39:55.177788+0000","last_active":"2024-03-16T09:40:00.065303+0000","last_peered":"2024-03-16T09:40:00.065303+0000","last_clean":"2024-03-16T09:40:00.065303+0000","last_became_active":"2024-03-16T09:39:55.177443+0000","last_became_peered":"2024-03-16T09:39:55.177443+0000","last_unstale":"2024-03-16T09:40:00.065303+0000","last_undegraded":"2024-03-16T09:40:00.065303+0000","last_fullsized":"2024-03-16T09:40:00.065303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:39.221712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4018,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191241+0000","last_change":"2024-03-16T09:39:56.178976+0000","last_active":"2024-03-16T09:39:57.191241+0000","last_peered":"2024-03-16T09:39:57.191241+0000","last_clean":"2024-03-16T09:39:57.191241+0000","last_became_active":"2024-03-16T09:39:56.178455+0000","last_became_peered":"2024-03-16T09:39:56.178455+0000","last_unstale":"2024-03-16T09:39:57.191241+0000","last_undegraded":"2024-03-16T09:39:57.191241+0000","last_fullsized":"2024-03-16T09:39:57.191241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:52:08.064792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731610+0000","last_change":"2024-03-16T09:39:55.182579+0000","last_active":"2024-03-16T09:39:58.731610+0000","last_peered":"2024-03-16T09:39:58.731610+0000","last_clean":"2024-03-16T09:39:58.731610+0000","last_became_active":"2024-03-16T09:39:55.181298+0000","last_became_peered":"2024-03-16T09:39:55.181298+0000","last_unstale":"2024-03-16T09:39:58.731610+0000","last_undegraded":"2024-03-16T09:39:58.731610+0000","last_fullsized":"2024-03-16T09:39:58.731610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:35.768812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268361+0000","last_change":"2024-03-16T09:39:56.188519+0000","last_active":"2024-03-16T09:39:57.268361+0000","last_peered":"2024-03-16T09:39:57.268361+0000","last_clean":"2024-03-16T09:39:57.268361+0000","last_became_active":"2024-03-16T09:39:56.188031+0000","last_became_peered":"2024-03-16T09:39:56.188031+0000","last_unstale":"2024-03-16T09:39:57.268361+0000","last_undegraded":"2024-03-16T09:39:57.268361+0000","last_fullsized":"2024-03-16T09:39:57.268361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:20:31.765657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.172284+0000","last_change":"2024-03-16T09:39:55.177523+0000","last_active":"2024-03-16T09:39:56.172284+0000","last_peered":"2024-03-16T09:39:56.172284+0000","last_clean":"2024-03-16T09:39:56.172284+0000","last_became_active":"2024-03-16T09:39:55.177143+0000","last_became_peered":"2024-03-16T09:39:55.177143+0000","last_unstale":"2024-03-16T09:39:56.172284+0000","last_undegraded":"2024-03-16T09:39:56.172284+0000","last_fullsized":"2024-03-16T09:39:56.172284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:05.357611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269548+0000","last_change":"2024-03-16T09:39:56.196794+0000","last_active":"2024-03-16T09:39:57.269548+0000","last_peered":"2024-03-16T09:39:57.269548+0000","last_clean":"2024-03-16T09:39:57.269548+0000","last_became_active":"2024-03-16T09:39:56.196425+0000","last_became_peered":"2024-03-16T09:39:56.196425+0000","last_unstale":"2024-03-16T09:39:57.269548+0000","last_undegraded":"2024-03-16T09:39:57.269548+0000","last_fullsized":"2024-03-16T09:39:57.269548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:26:45.952779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.735522+0000","last_change":"2024-03-16T09:39:55.175419+0000","last_active":"2024-03-16T09:39:58.735522+0000","last_peered":"2024-03-16T09:39:58.735522+0000","last_clean":"2024-03-16T09:39:58.735522+0000","last_became_active":"2024-03-16T09:39:55.175126+0000","last_became_peered":"2024-03-16T09:39:55.175126+0000","last_unstale":"2024-03-16T09:39:58.735522+0000","last_undegraded":"2024-03-16T09:39:58.735522+0000","last_fullsized":"2024-03-16T09:39:58.735522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:11.696911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269215+0000","last_change":"2024-03-16T09:39:56.176714+0000","last_active":"2024-03-16T09:39:57.269215+0000","last_peered":"2024-03-16T09:39:57.269215+0000","last_clean":"2024-03-16T09:39:57.269215+0000","last_became_active":"2024-03-16T09:39:56.176245+0000","last_became_peered":"2024-03-16T09:39:56.176245+0000","last_unstale":"2024-03-16T09:39:57.269215+0000","last_undegraded":"2024-03-16T09:39:57.269215+0000","last_fullsized":"2024-03-16T09:39:57.269215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:32.430207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.733635+0000","last_change":"2024-03-16T09:39:55.181486+0000","last_active":"2024-03-16T09:39:58.733635+0000","last_peered":"2024-03-16T09:39:58.733635+0000","last_clean":"2024-03-16T09:39:58.733635+0000","last_became_active":"2024-03-16T09:39:55.180128+0000","last_became_peered":"2024-03-16T09:39:55.180128+0000","last_unstale":"2024-03-16T09:39:58.733635+0000","last_undegraded":"2024-03-16T09:39:58.733635+0000","last_fullsized":"2024-03-16T09:39:58.733635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:45.323870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.199006+0000","last_change":"2024-03-16T09:39:56.196692+0000","last_active":"2024-03-16T09:39:56.199006+0000","last_peered":"2024-03-16T09:39:56.199006+0000","last_clean":"2024-03-16T09:39:56.199006+0000","last_became_active":"2024-03-16T09:39:56.196403+0000","last_became_peered":"2024-03-16T09:39:56.196403+0000","last_unstale":"2024-03-16T09:39:56.199006+0000","last_undegraded":"2024-03-16T09:39:56.199006+0000","last_fullsized":"2024-03-16T09:39:56.199006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:16.953824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:59.728978+0000","last_change":"2024-03-16T09:39:55.184156+0000","last_active":"2024-03-16T09:39:59.728978+0000","last_peered":"2024-03-16T09:39:59.728978+0000","last_clean":"2024-03-16T09:39:59.728978+0000","last_became_active":"2024-03-16T09:39:55.183563+0000","last_became_peered":"2024-03-16T09:39:55.183563+0000","last_unstale":"2024-03-16T09:39:59.728978+0000","last_undegraded":"2024-03-16T09:39:59.728978+0000","last_fullsized":"2024-03-16T09:39:59.728978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:22.376434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269043+0000","last_change":"2024-03-16T09:39:56.196699+0000","last_active":"2024-03-16T09:39:57.269043+0000","last_peered":"2024-03-16T09:39:57.269043+0000","last_clean":"2024-03-16T09:39:57.269043+0000","last_became_active":"2024-03-16T09:39:56.196386+0000","last_became_peered":"2024-03-16T09:39:56.196386+0000","last_unstale":"2024-03-16T09:39:57.269043+0000","last_undegraded":"2024-03-16T09:39:57.269043+0000","last_fullsized":"2024-03-16T09:39:57.269043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:20:32.734352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.188964+0000","last_change":"2024-03-16T09:39:55.175666+0000","last_active":"2024-03-16T09:39:57.188964+0000","last_peered":"2024-03-16T09:39:57.188964+0000","last_clean":"2024-03-16T09:39:57.188964+0000","last_became_active":"2024-03-16T09:39:55.175337+0000","last_became_peered":"2024-03-16T09:39:55.175337+0000","last_unstale":"2024-03-16T09:39:57.188964+0000","last_undegraded":"2024-03-16T09:39:57.188964+0000","last_fullsized":"2024-03-16T09:39:57.188964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:45:13.935669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268876+0000","last_change":"2024-03-16T09:39:56.184878+0000","last_active":"2024-03-16T09:39:57.268876+0000","last_peered":"2024-03-16T09:39:57.268876+0000","last_clean":"2024-03-16T09:39:57.268876+0000","last_became_active":"2024-03-16T09:39:56.184480+0000","last_became_peered":"2024-03-16T09:39:56.184480+0000","last_unstale":"2024-03-16T09:39:57.268876+0000","last_undegraded":"2024-03-16T09:39:57.268876+0000","last_fullsized":"2024-03-16T09:39:57.268876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:40:01.286993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269514+0000","last_change":"2024-03-16T09:39:55.179189+0000","last_active":"2024-03-16T09:39:57.269514+0000","last_peered":"2024-03-16T09:39:57.269514+0000","last_clean":"2024-03-16T09:39:57.269514+0000","last_became_active":"2024-03-16T09:39:55.178859+0000","last_became_peered":"2024-03-16T09:39:55.178859+0000","last_unstale":"2024-03-16T09:39:57.269514+0000","last_undegraded":"2024-03-16T09:39:57.269514+0000","last_fullsized":"2024-03-16T09:39:57.269514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:54:15.498386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269147+0000","last_change":"2024-03-16T09:39:56.200665+0000","last_active":"2024-03-16T09:39:57.269147+0000","last_peered":"2024-03-16T09:39:57.269147+0000","last_clean":"2024-03-16T09:39:57.269147+0000","last_became_active":"2024-03-16T09:39:56.200326+0000","last_became_peered":"2024-03-16T09:39:56.200326+0000","last_unstale":"2024-03-16T09:39:57.269147+0000","last_undegraded":"2024-03-16T09:39:57.269147+0000","last_fullsized":"2024-03-16T09:39:57.269147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:34.778630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.184330+0000","last_change":"2024-03-16T09:39:56.182670+0000","last_active":"2024-03-16T09:39:56.184330+0000","last_peered":"2024-03-16T09:39:56.184330+0000","last_clean":"2024-03-16T09:39:56.184330+0000","last_became_active":"2024-03-16T09:39:56.182009+0000","last_became_peered":"2024-03-16T09:39:56.182009+0000","last_unstale":"2024-03-16T09:39:56.184330+0000","last_undegraded":"2024-03-16T09:39:56.184330+0000","last_fullsized":"2024-03-16T09:39:56.184330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:02:32.399218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:59.735655+0000","last_change":"2024-03-16T09:39:55.171609+0000","last_active":"2024-03-16T09:39:59.735655+0000","last_peered":"2024-03-16T09:39:59.735655+0000","last_clean":"2024-03-16T09:39:59.735655+0000","last_became_active":"2024-03-16T09:39:55.171301+0000","last_became_peered":"2024-03-16T09:39:55.171301+0000","last_unstale":"2024-03-16T09:39:59.735655+0000","last_undegraded":"2024-03-16T09:39:59.735655+0000","last_fullsized":"2024-03-16T09:39:59.735655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:28.265342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185679+0000","last_change":"2024-03-16T09:39:56.190270+0000","last_active":"2024-03-16T09:39:57.185679+0000","last_peered":"2024-03-16T09:39:57.185679+0000","last_clean":"2024-03-16T09:39:57.185679+0000","last_became_active":"2024-03-16T09:39:56.189882+0000","last_became_peered":"2024-03-16T09:39:56.189882+0000","last_unstale":"2024-03-16T09:39:57.185679+0000","last_undegraded":"2024-03-16T09:39:57.185679+0000","last_fullsized":"2024-03-16T09:39:57.185679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:47.290540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731782+0000","last_change":"2024-03-16T09:39:55.172824+0000","last_active":"2024-03-16T09:39:58.731782+0000","last_peered":"2024-03-16T09:39:58.731782+0000","last_clean":"2024-03-16T09:39:58.731782+0000","last_became_active":"2024-03-16T09:39:55.172493+0000","last_became_peered":"2024-03-16T09:39:55.172493+0000","last_unstale":"2024-03-16T09:39:58.731782+0000","last_undegraded":"2024-03-16T09:39:58.731782+0000","last_fullsized":"2024-03-16T09:39:58.731782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:39.188176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269322+0000","last_change":"2024-03-16T09:38:43.693644+0000","last_active":"2024-03-16T09:39:57.269322+0000","last_peered":"2024-03-16T09:39:57.269322+0000","last_clean":"2024-03-16T09:39:57.269322+0000","last_became_active":"2024-03-16T09:38:43.693202+0000","last_became_peered":"2024-03-16T09:38:43.693202+0000","last_unstale":"2024-03-16T09:39:57.269322+0000","last_undegraded":"2024-03-16T09:39:57.269322+0000","last_fullsized":"2024-03-16T09:39:57.269322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_clean_scrub_stamp":"2024-03-16T09:35:24.153508+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:43:06.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268148+0000","last_change":"2024-03-16T09:39:56.197109+0000","last_active":"2024-03-16T09:39:57.268148+0000","last_peered":"2024-03-16T09:39:57.268148+0000","last_clean":"2024-03-16T09:39:57.268148+0000","last_became_active":"2024-03-16T09:39:56.196775+0000","last_became_peered":"2024-03-16T09:39:56.196775+0000","last_unstale":"2024-03-16T09:39:57.268148+0000","last_undegraded":"2024-03-16T09:39:57.268148+0000","last_fullsized":"2024-03-16T09:39:57.268148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:29:10.401423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732143+0000","last_change":"2024-03-16T09:39:55.175744+0000","last_active":"2024-03-16T09:39:58.732143+0000","last_peered":"2024-03-16T09:39:58.732143+0000","last_clean":"2024-03-16T09:39:58.732143+0000","last_became_active":"2024-03-16T09:39:55.175445+0000","last_became_peered":"2024-03-16T09:39:55.175445+0000","last_unstale":"2024-03-16T09:39:58.732143+0000","last_undegraded":"2024-03-16T09:39:58.732143+0000","last_fullsized":"2024-03-16T09:39:58.732143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:39.494340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713098+0000","last_change":"2024-03-16T09:39:55.183441+0000","last_active":"2024-03-16T09:39:57.713098+0000","last_peered":"2024-03-16T09:39:57.713098+0000","last_clean":"2024-03-16T09:39:57.713098+0000","last_became_active":"2024-03-16T09:39:55.183115+0000","last_became_peered":"2024-03-16T09:39:55.183115+0000","last_unstale":"2024-03-16T09:39:57.713098+0000","last_undegraded":"2024-03-16T09:39:57.713098+0000","last_fullsized":"2024-03-16T09:39:57.713098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:55:31.525201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268618+0000","last_change":"2024-03-16T09:39:56.179205+0000","last_active":"2024-03-16T09:39:57.268618+0000","last_peered":"2024-03-16T09:39:57.268618+0000","last_clean":"2024-03-16T09:39:57.268618+0000","last_became_active":"2024-03-16T09:39:56.178849+0000","last_became_peered":"2024-03-16T09:39:56.178849+0000","last_unstale":"2024-03-16T09:39:57.268618+0000","last_undegraded":"2024-03-16T09:39:57.268618+0000","last_fullsized":"2024-03-16T09:39:57.268618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:10.016099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:40:00.188079+0000","last_change":"2024-03-16T09:39:55.180859+0000","last_active":"2024-03-16T09:40:00.188079+0000","last_peered":"2024-03-16T09:40:00.188079+0000","last_clean":"2024-03-16T09:40:00.188079+0000","last_became_active":"2024-03-16T09:39:55.180514+0000","last_became_peered":"2024-03-16T09:39:55.180514+0000","last_unstale":"2024-03-16T09:40:00.188079+0000","last_undegraded":"2024-03-16T09:40:00.188079+0000","last_fullsized":"2024-03-16T09:40:00.188079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:46.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185799+0000","last_change":"2024-03-16T09:39:56.187626+0000","last_active":"2024-03-16T09:39:57.185799+0000","last_peered":"2024-03-16T09:39:57.185799+0000","last_clean":"2024-03-16T09:39:57.185799+0000","last_became_active":"2024-03-16T09:39:56.187284+0000","last_became_peered":"2024-03-16T09:39:56.187284+0000","last_unstale":"2024-03-16T09:39:57.185799+0000","last_undegraded":"2024-03-16T09:39:57.185799+0000","last_fullsized":"2024-03-16T09:39:57.185799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:23.511603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713140+0000","last_change":"2024-03-16T09:39:55.172912+0000","last_active":"2024-03-16T09:39:57.713140+0000","last_peered":"2024-03-16T09:39:57.713140+0000","last_clean":"2024-03-16T09:39:57.713140+0000","last_became_active":"2024-03-16T09:39:55.172592+0000","last_became_peered":"2024-03-16T09:39:55.172592+0000","last_unstale":"2024-03-16T09:39:57.713140+0000","last_undegraded":"2024-03-16T09:39:57.713140+0000","last_fullsized":"2024-03-16T09:39:57.713140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:24.344588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268563+0000","last_change":"2024-03-16T09:39:56.197652+0000","last_active":"2024-03-16T09:39:57.268563+0000","last_peered":"2024-03-16T09:39:57.268563+0000","last_clean":"2024-03-16T09:39:57.268563+0000","last_became_active":"2024-03-16T09:39:56.197230+0000","last_became_peered":"2024-03-16T09:39:56.197230+0000","last_unstale":"2024-03-16T09:39:57.268563+0000","last_undegraded":"2024-03-16T09:39:57.268563+0000","last_fullsized":"2024-03-16T09:39:57.268563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:45:08.661372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.715034+0000","last_change":"2024-03-16T09:39:55.175140+0000","last_active":"2024-03-16T09:39:57.715034+0000","last_peered":"2024-03-16T09:39:57.715034+0000","last_clean":"2024-03-16T09:39:57.715034+0000","last_became_active":"2024-03-16T09:39:55.174764+0000","last_became_peered":"2024-03-16T09:39:55.174764+0000","last_unstale":"2024-03-16T09:39:57.715034+0000","last_undegraded":"2024-03-16T09:39:57.715034+0000","last_fullsized":"2024-03-16T09:39:57.715034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:03:04.591466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191248+0000","last_change":"2024-03-16T09:39:56.179326+0000","last_active":"2024-03-16T09:39:57.191248+0000","last_peered":"2024-03-16T09:39:57.191248+0000","last_clean":"2024-03-16T09:39:57.191248+0000","last_became_active":"2024-03-16T09:39:56.178681+0000","last_became_peered":"2024-03-16T09:39:56.178681+0000","last_unstale":"2024-03-16T09:39:57.191248+0000","last_undegraded":"2024-03-16T09:39:57.191248+0000","last_fullsized":"2024-03-16T09:39:57.191248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:26.137807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713555+0000","last_change":"2024-03-16T09:39:55.173820+0000","last_active":"2024-03-16T09:39:57.713555+0000","last_peered":"2024-03-16T09:39:57.713555+0000","last_clean":"2024-03-16T09:39:57.713555+0000","last_became_active":"2024-03-16T09:39:55.173472+0000","last_became_peered":"2024-03-16T09:39:55.173472+0000","last_unstale":"2024-03-16T09:39:57.713555+0000","last_undegraded":"2024-03-16T09:39:57.713555+0000","last_fullsized":"2024-03-16T09:39:57.713555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:14.082367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268901+0000","last_change":"2024-03-16T09:39:56.178535+0000","last_active":"2024-03-16T09:39:57.268901+0000","last_peered":"2024-03-16T09:39:57.268901+0000","last_clean":"2024-03-16T09:39:57.268901+0000","last_became_active":"2024-03-16T09:39:56.178210+0000","last_became_peered":"2024-03-16T09:39:56.178210+0000","last_unstale":"2024-03-16T09:39:57.268901+0000","last_undegraded":"2024-03-16T09:39:57.268901+0000","last_fullsized":"2024-03-16T09:39:57.268901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:18.680469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268109+0000","last_change":"2024-03-16T09:39:55.183074+0000","last_active":"2024-03-16T09:39:57.268109+0000","last_peered":"2024-03-16T09:39:57.268109+0000","last_clean":"2024-03-16T09:39:57.268109+0000","last_became_active":"2024-03-16T09:39:55.182695+0000","last_became_peered":"2024-03-16T09:39:55.182695+0000","last_unstale":"2024-03-16T09:39:57.268109+0000","last_undegraded":"2024-03-16T09:39:57.268109+0000","last_fullsized":"2024-03-16T09:39:57.268109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:51.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185745+0000","last_change":"2024-03-16T09:39:56.193434+0000","last_active":"2024-03-16T09:39:57.185745+0000","last_peered":"2024-03-16T09:39:57.185745+0000","last_clean":"2024-03-16T09:39:57.185745+0000","last_became_active":"2024-03-16T09:39:56.193051+0000","last_became_peered":"2024-03-16T09:39:56.193051+0000","last_unstale":"2024-03-16T09:39:57.185745+0000","last_undegraded":"2024-03-16T09:39:57.185745+0000","last_fullsized":"2024-03-16T09:39:57.185745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:12:26.767117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.712591+0000","last_change":"2024-03-16T09:39:55.179916+0000","last_active":"2024-03-16T09:39:57.712591+0000","last_peered":"2024-03-16T09:39:57.712591+0000","last_clean":"2024-03-16T09:39:57.712591+0000","last_became_active":"2024-03-16T09:39:55.179635+0000","last_became_peered":"2024-03-16T09:39:55.179635+0000","last_unstale":"2024-03-16T09:39:57.712591+0000","last_undegraded":"2024-03-16T09:39:57.712591+0000","last_fullsized":"2024-03-16T09:39:57.712591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:08.955039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269687+0000","last_change":"2024-03-16T09:39:56.194044+0000","last_active":"2024-03-16T09:39:57.269687+0000","last_peered":"2024-03-16T09:39:57.269687+0000","last_clean":"2024-03-16T09:39:57.269687+0000","last_became_active":"2024-03-16T09:39:56.193490+0000","last_became_peered":"2024-03-16T09:39:56.193490+0000","last_unstale":"2024-03-16T09:39:57.269687+0000","last_undegraded":"2024-03-16T09:39:57.269687+0000","last_fullsized":"2024-03-16T09:39:57.269687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:19:59.916892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732272+0000","last_change":"2024-03-16T09:39:55.182652+0000","last_active":"2024-03-16T09:39:58.732272+0000","last_peered":"2024-03-16T09:39:58.732272+0000","last_clean":"2024-03-16T09:39:58.732272+0000","last_became_active":"2024-03-16T09:39:55.181432+0000","last_became_peered":"2024-03-16T09:39:55.181432+0000","last_unstale":"2024-03-16T09:39:58.732272+0000","last_undegraded":"2024-03-16T09:39:58.732272+0000","last_fullsized":"2024-03-16T09:39:58.732272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:34.959845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.270027+0000","last_change":"2024-03-16T09:39:56.188970+0000","last_active":"2024-03-16T09:39:57.270027+0000","last_peered":"2024-03-16T09:39:57.270027+0000","last_clean":"2024-03-16T09:39:57.270027+0000","last_became_active":"2024-03-16T09:39:56.188607+0000","last_became_peered":"2024-03-16T09:39:56.188607+0000","last_unstale":"2024-03-16T09:39:57.270027+0000","last_undegraded":"2024-03-16T09:39:57.270027+0000","last_fullsized":"2024-03-16T09:39:57.270027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:39:24.532114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.195943+0000","last_change":"2024-03-16T09:39:55.181908+0000","last_active":"2024-03-16T09:39:56.195943+0000","last_peered":"2024-03-16T09:39:56.195943+0000","last_clean":"2024-03-16T09:39:56.195943+0000","last_became_active":"2024-03-16T09:39:55.181055+0000","last_became_peered":"2024-03-16T09:39:55.181055+0000","last_unstale":"2024-03-16T09:39:56.195943+0000","last_undegraded":"2024-03-16T09:39:56.195943+0000","last_fullsized":"2024-03-16T09:39:56.195943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:38.256664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269617+0000","last_change":"2024-03-16T09:39:56.189743+0000","last_active":"2024-03-16T09:39:57.269617+0000","last_peered":"2024-03-16T09:39:57.269617+0000","last_clean":"2024-03-16T09:39:57.269617+0000","last_became_active":"2024-03-16T09:39:56.189339+0000","last_became_peered":"2024-03-16T09:39:56.189339+0000","last_unstale":"2024-03-16T09:39:57.269617+0000","last_undegraded":"2024-03-16T09:39:57.269617+0000","last_fullsized":"2024-03-16T09:39:57.269617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:54.694191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268125+0000","last_change":"2024-03-16T09:39:55.182357+0000","last_active":"2024-03-16T09:39:57.268125+0000","last_peered":"2024-03-16T09:39:57.268125+0000","last_clean":"2024-03-16T09:39:57.268125+0000","last_became_active":"2024-03-16T09:39:55.182053+0000","last_became_peered":"2024-03-16T09:39:55.182053+0000","last_unstale":"2024-03-16T09:39:57.268125+0000","last_undegraded":"2024-03-16T09:39:57.268125+0000","last_fullsized":"2024-03-16T09:39:57.268125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:40.735596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269683+0000","last_change":"2024-03-16T09:39:56.181636+0000","last_active":"2024-03-16T09:39:57.269683+0000","last_peered":"2024-03-16T09:39:57.269683+0000","last_clean":"2024-03-16T09:39:57.269683+0000","last_became_active":"2024-03-16T09:39:56.181309+0000","last_became_peered":"2024-03-16T09:39:56.181309+0000","last_unstale":"2024-03-16T09:39:57.269683+0000","last_undegraded":"2024-03-16T09:39:57.269683+0000","last_fullsized":"2024-03-16T09:39:57.269683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:36.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268746+0000","last_change":"2024-03-16T09:39:56.186640+0000","last_active":"2024-03-16T09:39:57.268746+0000","last_peered":"2024-03-16T09:39:57.268746+0000","last_clean":"2024-03-16T09:39:57.268746+0000","last_became_active":"2024-03-16T09:39:56.186311+0000","last_became_peered":"2024-03-16T09:39:56.186311+0000","last_unstale":"2024-03-16T09:39:57.268746+0000","last_undegraded":"2024-03-16T09:39:57.268746+0000","last_fullsized":"2024-03-16T09:39:57.268746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:58:44.451068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268563+0000","last_change":"2024-03-16T09:39:56.196887+0000","last_active":"2024-03-16T09:39:57.268563+0000","last_peered":"2024-03-16T09:39:57.268563+0000","last_clean":"2024-03-16T09:39:57.268563+0000","last_became_active":"2024-03-16T09:39:56.196568+0000","last_became_peered":"2024-03-16T09:39:56.196568+0000","last_unstale":"2024-03-16T09:39:57.268563+0000","last_undegraded":"2024-03-16T09:39:57.268563+0000","last_fullsized":"2024-03-16T09:39:57.268563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:08:35.577630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268213+0000","last_change":"2024-03-16T09:39:56.186394+0000","last_active":"2024-03-16T09:39:57.268213+0000","last_peered":"2024-03-16T09:39:57.268213+0000","last_clean":"2024-03-16T09:39:57.268213+0000","last_became_active":"2024-03-16T09:39:56.186053+0000","last_became_peered":"2024-03-16T09:39:56.186053+0000","last_unstale":"2024-03-16T09:39:57.268213+0000","last_undegraded":"2024-03-16T09:39:57.268213+0000","last_fullsized":"2024-03-16T09:39:57.268213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:41.529601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268396+0000","last_change":"2024-03-16T09:39:56.197622+0000","last_active":"2024-03-16T09:39:57.268396+0000","last_peered":"2024-03-16T09:39:57.268396+0000","last_clean":"2024-03-16T09:39:57.268396+0000","last_became_active":"2024-03-16T09:39:56.196885+0000","last_became_peered":"2024-03-16T09:39:56.196885+0000","last_unstale":"2024-03-16T09:39:57.268396+0000","last_undegraded":"2024-03-16T09:39:57.268396+0000","last_fullsized":"2024-03-16T09:39:57.268396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:08:50.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268461+0000","last_change":"2024-03-16T09:39:56.192318+0000","last_active":"2024-03-16T09:39:57.268461+0000","last_peered":"2024-03-16T09:39:57.268461+0000","last_clean":"2024-03-16T09:39:57.268461+0000","last_became_active":"2024-03-16T09:39:56.191998+0000","last_became_peered":"2024-03-16T09:39:56.191998+0000","last_unstale":"2024-03-16T09:39:57.268461+0000","last_undegraded":"2024-03-16T09:39:57.268461+0000","last_fullsized":"2024-03-16T09:39:57.268461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:51:47.025415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268942+0000","last_change":"2024-03-16T09:39:56.181262+0000","last_active":"2024-03-16T09:39:57.268942+0000","last_peered":"2024-03-16T09:39:57.268942+0000","last_clean":"2024-03-16T09:39:57.268942+0000","last_became_active":"2024-03-16T09:39:56.180916+0000","last_became_peered":"2024-03-16T09:39:56.180916+0000","last_unstale":"2024-03-16T09:39:57.268942+0000","last_undegraded":"2024-03-16T09:39:57.268942+0000","last_fullsized":"2024-03-16T09:39:57.268942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:34:45.056025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268834+0000","last_change":"2024-03-16T09:39:56.176621+0000","last_active":"2024-03-16T09:39:57.268834+0000","last_peered":"2024-03-16T09:39:57.268834+0000","last_clean":"2024-03-16T09:39:57.268834+0000","last_became_active":"2024-03-16T09:39:56.176296+0000","last_became_peered":"2024-03-16T09:39:56.176296+0000","last_unstale":"2024-03-16T09:39:57.268834+0000","last_undegraded":"2024-03-16T09:39:57.268834+0000","last_fullsized":"2024-03-16T09:39:57.268834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:54.673937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185552+0000","last_change":"2024-03-16T09:39:56.197689+0000","last_active":"2024-03-16T09:39:57.185552+0000","last_peered":"2024-03-16T09:39:57.185552+0000","last_clean":"2024-03-16T09:39:57.185552+0000","last_became_active":"2024-03-16T09:39:56.197343+0000","last_became_peered":"2024-03-16T09:39:56.197343+0000","last_unstale":"2024-03-16T09:39:57.185552+0000","last_undegraded":"2024-03-16T09:39:57.185552+0000","last_fullsized":"2024-03-16T09:39:57.185552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:28:10.218048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.194276+0000","last_change":"2024-03-16T09:39:56.193551+0000","last_active":"2024-03-16T09:39:56.194276+0000","last_peered":"2024-03-16T09:39:56.194276+0000","last_clean":"2024-03-16T09:39:56.194276+0000","last_became_active":"2024-03-16T09:39:56.192901+0000","last_became_peered":"2024-03-16T09:39:56.192901+0000","last_unstale":"2024-03-16T09:39:56.194276+0000","last_undegraded":"2024-03-16T09:39:56.194276+0000","last_fullsized":"2024-03-16T09:39:56.194276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:27.116485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185283+0000","last_change":"2024-03-16T09:39:56.188484+0000","last_active":"2024-03-16T09:39:57.185283+0000","last_peered":"2024-03-16T09:39:57.185283+0000","last_clean":"2024-03-16T09:39:57.185283+0000","last_became_active":"2024-03-16T09:39:56.188144+0000","last_became_peered":"2024-03-16T09:39:56.188144+0000","last_unstale":"2024-03-16T09:39:57.185283+0000","last_undegraded":"2024-03-16T09:39:57.185283+0000","last_fullsized":"2024-03-16T09:39:57.185283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:01:10.042765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.181936+0000","last_change":"2024-03-16T09:39:56.187448+0000","last_active":"2024-03-16T09:39:57.181936+0000","last_peered":"2024-03-16T09:39:57.181936+0000","last_clean":"2024-03-16T09:39:57.181936+0000","last_became_active":"2024-03-16T09:39:56.186772+0000","last_became_peered":"2024-03-16T09:39:56.186772+0000","last_unstale":"2024-03-16T09:39:57.181936+0000","last_undegraded":"2024-03-16T09:39:57.181936+0000","last_fullsized":"2024-03-16T09:39:57.181936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:45:43.315011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268500+0000","last_change":"2024-03-16T09:39:56.191004+0000","last_active":"2024-03-16T09:39:57.268500+0000","last_peered":"2024-03-16T09:39:57.268500+0000","last_clean":"2024-03-16T09:39:57.268500+0000","last_became_active":"2024-03-16T09:39:56.190646+0000","last_became_peered":"2024-03-16T09:39:56.190646+0000","last_unstale":"2024-03-16T09:39:57.268500+0000","last_undegraded":"2024-03-16T09:39:57.268500+0000","last_fullsized":"2024-03-16T09:39:57.268500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:03.191129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268901+0000","last_change":"2024-03-16T09:39:56.197552+0000","last_active":"2024-03-16T09:39:57.268901+0000","last_peered":"2024-03-16T09:39:57.268901+0000","last_clean":"2024-03-16T09:39:57.268901+0000","last_became_active":"2024-03-16T09:39:56.197215+0000","last_became_peered":"2024-03-16T09:39:56.197215+0000","last_unstale":"2024-03-16T09:39:57.268901+0000","last_undegraded":"2024-03-16T09:39:57.268901+0000","last_fullsized":"2024-03-16T09:39:57.268901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:45.795328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6424,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":124,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":16106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":913408,"data_stored":434445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1036573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":443833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":443753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1540096,"data_stored":1035292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462949,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":445283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":933888,"data_stored":443119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":443037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724587,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36832,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708320,"statfs":{"total":95995035648,"available":95957319680,"internally_reserved":0,"allocated":1519616,"data_stored":1024995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T09:40:02.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 -- 172.21.15.77:0/3690289273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 msgr2=0x7fe450101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 --2- 172.21.15.77:0/3690289273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450101c20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe440009900 tx=0x7fe44002bc30 comp rx=0 tx=0).stop 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 -- 172.21.15.77:0/3690289273 shutdown_connections 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 --2- 172.21.15.77:0/3690289273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4501033b0 0x7fe450109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 --2- 172.21.15.77:0/3690289273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4501029f0 0x7fe450102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 --2- 172.21.15.77:0/3690289273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450101c20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 -- 172.21.15.77:0/3690289273 >> 172.21.15.77:0/3690289273 conn(0x7fe4500fcd90 msgr2=0x7fe4500ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 -- 172.21.15.77:0/3690289273 shutdown_connections 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 -- 172.21.15.77:0/3690289273 wait complete. 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 Processor -- start 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.569+0000 7fe456826640 1 -- start start 2024-03-16T09:40:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe456826640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe456826640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4501029f0 0x7fe45019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe456826640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4501033b0 0x7fe4501a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe456826640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe45010d320 con 0x7fe450101800 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe456826640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe45010d1a0 con 0x7fe4501029f0 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe456826640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe45010d4a0 con 0x7fe4501033b0 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59798/0 (socket says 172.21.15.77:59798) 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 -- 172.21.15.77:0/1779840208 learned_addr learned my addr 172.21.15.77:0/1779840208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44f7fe640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4501029f0 0x7fe45019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe454dc3640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4501033b0 0x7fe4501a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 -- 172.21.15.77:0/1779840208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4501033b0 msgr2=0x7fe4501a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4501033b0 0x7fe4501a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 -- 172.21.15.77:0/1779840208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4501029f0 msgr2=0x7fe45019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4501029f0 0x7fe45019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 -- 172.21.15.77:0/1779840208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4501a1890 con 0x7fe450101800 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44ffff640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450199c80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe440000c00 tx=0x7fe440005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44d7fa640 1 -- 172.21.15.77:0/1779840208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440043020 con 0x7fe450101800 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe44d7fa640 1 -- 172.21.15.77:0/1779840208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe440004350 con 0x7fe450101800 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.570+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4501a1b20 con 0x7fe450101800 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.571+0000 7fe44d7fa640 1 -- 172.21.15.77:0/1779840208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440003b80 con 0x7fe450101800 2024-03-16T09:40:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.571+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4501a2070 con 0x7fe450101800 2024-03-16T09:40:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.572+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4501a2240 con 0x7fe450101800 2024-03-16T09:40:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.572+0000 7fe44d7fa640 1 -- 172.21.15.77:0/1779840208 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe440046020 con 0x7fe450101800 2024-03-16T09:40:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.573+0000 7fe44d7fa640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4380a2890 0x7fe4380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.573+0000 7fe44f7fe640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4380a2890 0x7fe4380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.573+0000 7fe44d7fa640 1 -- 172.21.15.77:0/1779840208 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe4400f0ae0 con 0x7fe450101800 2024-03-16T09:40:02.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.573+0000 7fe44f7fe640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4380a2890 0x7fe4380a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe444009bd0 tx=0x7fe444005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:02.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.577+0000 7fe44d7fa640 1 -- 172.21.15.77:0/1779840208 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe44004d020 con 0x7fe450101800 2024-03-16T09:40:02.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.753+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe450075130 con 0x7fe4380a2890 2024-03-16T09:40:02.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.761+0000 7fe44d7fa640 1 -- 172.21.15.77:0/1779840208 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258768 (secure 0 0 0) 0x7fe450075130 con 0x7fe4380a2890 2024-03-16T09:40:02.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:40:02.768 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T09:40:02.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4380a2890 msgr2=0x7fe4380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4380a2890 0x7fe4380a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe444009bd0 tx=0x7fe444005f90 comp rx=0 tx=0).stop 2024-03-16T09:40:02.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 msgr2=0x7fe450199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:02.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450199c80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe440000c00 tx=0x7fe440005dc0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 shutdown_connections 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe4380a2890 0x7fe4380a4d50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4501033b0 0x7fe4501a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4501029f0 0x7fe45019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 --2- 172.21.15.77:0/1779840208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe450101800 0x7fe450199c80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 >> 172.21.15.77:0/1779840208 conn(0x7fe4500fcd90 msgr2=0x7fe450073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 shutdown_connections 2024-03-16T09:40:02.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:02.764+0000 7fe456826640 1 -- 172.21.15.77:0/1779840208 wait complete. 2024-03-16T09:40:02.778 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-16T09:40:01.771713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596909,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":139,"num_read_kb":121,"num_write":306,"num_write_kb":2979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":238,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357816,"kb_used_data":12764,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584008,"statfs":{"total":1151940427776,"available":1151574024192,"internally_reserved":0,"allocated":13070336,"data_stored":7082672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6541,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":125,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.608347"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269281+0000","last_change":"2024-03-16T09:39:56.184389+0000","last_active":"2024-03-16T09:39:57.269281+0000","last_peered":"2024-03-16T09:39:57.269281+0000","last_clean":"2024-03-16T09:39:57.269281+0000","last_became_active":"2024-03-16T09:39:56.184129+0000","last_became_peered":"2024-03-16T09:39:56.184129+0000","last_unstale":"2024-03-16T09:39:57.269281+0000","last_undegraded":"2024-03-16T09:39:57.269281+0000","last_fullsized":"2024-03-16T09:39:57.269281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:33.786788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268573+0000","last_change":"2024-03-16T09:39:56.193451+0000","last_active":"2024-03-16T09:39:57.268573+0000","last_peered":"2024-03-16T09:39:57.268573+0000","last_clean":"2024-03-16T09:39:57.268573+0000","last_became_active":"2024-03-16T09:39:56.187381+0000","last_became_peered":"2024-03-16T09:39:56.187381+0000","last_unstale":"2024-03-16T09:39:57.268573+0000","last_undegraded":"2024-03-16T09:39:57.268573+0000","last_fullsized":"2024-03-16T09:39:57.268573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:51.915654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268387+0000","last_change":"2024-03-16T09:39:56.183785+0000","last_active":"2024-03-16T09:39:57.268387+0000","last_peered":"2024-03-16T09:39:57.268387+0000","last_clean":"2024-03-16T09:39:57.268387+0000","last_became_active":"2024-03-16T09:39:56.183421+0000","last_became_peered":"2024-03-16T09:39:56.183421+0000","last_unstale":"2024-03-16T09:39:57.268387+0000","last_undegraded":"2024-03-16T09:39:57.268387+0000","last_fullsized":"2024-03-16T09:39:57.268387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:47.259139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268934+0000","last_change":"2024-03-16T09:39:56.195743+0000","last_active":"2024-03-16T09:39:57.268934+0000","last_peered":"2024-03-16T09:39:57.268934+0000","last_clean":"2024-03-16T09:39:57.268934+0000","last_became_active":"2024-03-16T09:39:56.194818+0000","last_became_peered":"2024-03-16T09:39:56.194818+0000","last_unstale":"2024-03-16T09:39:57.268934+0000","last_undegraded":"2024-03-16T09:39:57.268934+0000","last_fullsized":"2024-03-16T09:39:57.268934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:39:43.123879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191373+0000","last_change":"2024-03-16T09:39:56.184970+0000","last_active":"2024-03-16T09:39:57.191373+0000","last_peered":"2024-03-16T09:39:57.191373+0000","last_clean":"2024-03-16T09:39:57.191373+0000","last_became_active":"2024-03-16T09:39:56.184632+0000","last_became_peered":"2024-03-16T09:39:56.184632+0000","last_unstale":"2024-03-16T09:39:57.191373+0000","last_undegraded":"2024-03-16T09:39:57.191373+0000","last_fullsized":"2024-03-16T09:39:57.191373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:46:51.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269044+0000","last_change":"2024-03-16T09:39:56.199688+0000","last_active":"2024-03-16T09:39:57.269044+0000","last_peered":"2024-03-16T09:39:57.269044+0000","last_clean":"2024-03-16T09:39:57.269044+0000","last_became_active":"2024-03-16T09:39:56.199094+0000","last_became_peered":"2024-03-16T09:39:56.199094+0000","last_unstale":"2024-03-16T09:39:57.269044+0000","last_undegraded":"2024-03-16T09:39:57.269044+0000","last_fullsized":"2024-03-16T09:39:57.269044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:10.855504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269892+0000","last_change":"2024-03-16T09:39:56.180816+0000","last_active":"2024-03-16T09:39:57.269892+0000","last_peered":"2024-03-16T09:39:57.269892+0000","last_clean":"2024-03-16T09:39:57.269892+0000","last_became_active":"2024-03-16T09:39:56.180499+0000","last_became_peered":"2024-03-16T09:39:56.180499+0000","last_unstale":"2024-03-16T09:39:57.269892+0000","last_undegraded":"2024-03-16T09:39:57.269892+0000","last_fullsized":"2024-03-16T09:39:57.269892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:32:09.444494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185191+0000","last_change":"2024-03-16T09:39:56.189960+0000","last_active":"2024-03-16T09:39:57.185191+0000","last_peered":"2024-03-16T09:39:57.185191+0000","last_clean":"2024-03-16T09:39:57.185191+0000","last_became_active":"2024-03-16T09:39:56.189609+0000","last_became_peered":"2024-03-16T09:39:56.189609+0000","last_unstale":"2024-03-16T09:39:57.185191+0000","last_undegraded":"2024-03-16T09:39:57.185191+0000","last_fullsized":"2024-03-16T09:39:57.185191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:50.219703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182394+0000","last_change":"2024-03-16T09:39:56.188383+0000","last_active":"2024-03-16T09:39:57.182394+0000","last_peered":"2024-03-16T09:39:57.182394+0000","last_clean":"2024-03-16T09:39:57.182394+0000","last_became_active":"2024-03-16T09:39:56.187926+0000","last_became_peered":"2024-03-16T09:39:56.187926+0000","last_unstale":"2024-03-16T09:39:57.182394+0000","last_undegraded":"2024-03-16T09:39:57.182394+0000","last_fullsized":"2024-03-16T09:39:57.182394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:06.422573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269391+0000","last_change":"2024-03-16T09:39:56.201584+0000","last_active":"2024-03-16T09:39:57.269391+0000","last_peered":"2024-03-16T09:39:57.269391+0000","last_clean":"2024-03-16T09:39:57.269391+0000","last_became_active":"2024-03-16T09:39:56.201320+0000","last_became_peered":"2024-03-16T09:39:56.201320+0000","last_unstale":"2024-03-16T09:39:57.269391+0000","last_undegraded":"2024-03-16T09:39:57.269391+0000","last_fullsized":"2024-03-16T09:39:57.269391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:18:00.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269927+0000","last_change":"2024-03-16T09:39:56.191125+0000","last_active":"2024-03-16T09:39:57.269927+0000","last_peered":"2024-03-16T09:39:57.269927+0000","last_clean":"2024-03-16T09:39:57.269927+0000","last_became_active":"2024-03-16T09:39:56.190795+0000","last_became_peered":"2024-03-16T09:39:56.190795+0000","last_unstale":"2024-03-16T09:39:57.269927+0000","last_undegraded":"2024-03-16T09:39:57.269927+0000","last_fullsized":"2024-03-16T09:39:57.269927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:44:10.894995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268723+0000","last_change":"2024-03-16T09:39:56.195652+0000","last_active":"2024-03-16T09:39:57.268723+0000","last_peered":"2024-03-16T09:39:57.268723+0000","last_clean":"2024-03-16T09:39:57.268723+0000","last_became_active":"2024-03-16T09:39:56.195270+0000","last_became_peered":"2024-03-16T09:39:56.195270+0000","last_unstale":"2024-03-16T09:39:57.268723+0000","last_undegraded":"2024-03-16T09:39:57.268723+0000","last_fullsized":"2024-03-16T09:39:57.268723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:03:41.546991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268370+0000","last_change":"2024-03-16T09:39:56.184622+0000","last_active":"2024-03-16T09:39:57.268370+0000","last_peered":"2024-03-16T09:39:57.268370+0000","last_clean":"2024-03-16T09:39:57.268370+0000","last_became_active":"2024-03-16T09:39:56.183532+0000","last_became_peered":"2024-03-16T09:39:56.183532+0000","last_unstale":"2024-03-16T09:39:57.268370+0000","last_undegraded":"2024-03-16T09:39:57.268370+0000","last_fullsized":"2024-03-16T09:39:57.268370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:11:24.574088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268685+0000","last_change":"2024-03-16T09:39:56.184393+0000","last_active":"2024-03-16T09:39:57.268685+0000","last_peered":"2024-03-16T09:39:57.268685+0000","last_clean":"2024-03-16T09:39:57.268685+0000","last_became_active":"2024-03-16T09:39:56.183783+0000","last_became_peered":"2024-03-16T09:39:56.183783+0000","last_unstale":"2024-03-16T09:39:57.268685+0000","last_undegraded":"2024-03-16T09:39:57.268685+0000","last_fullsized":"2024-03-16T09:39:57.268685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:16:22.131116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268036+0000","last_change":"2024-03-16T09:39:56.197365+0000","last_active":"2024-03-16T09:39:57.268036+0000","last_peered":"2024-03-16T09:39:57.268036+0000","last_clean":"2024-03-16T09:39:57.268036+0000","last_became_active":"2024-03-16T09:39:56.197020+0000","last_became_peered":"2024-03-16T09:39:56.197020+0000","last_unstale":"2024-03-16T09:39:57.268036+0000","last_undegraded":"2024-03-16T09:39:57.268036+0000","last_fullsized":"2024-03-16T09:39:57.268036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:29.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.181798+0000","last_change":"2024-03-16T09:39:56.185513+0000","last_active":"2024-03-16T09:39:57.181798+0000","last_peered":"2024-03-16T09:39:57.181798+0000","last_clean":"2024-03-16T09:39:57.181798+0000","last_became_active":"2024-03-16T09:39:56.184384+0000","last_became_peered":"2024-03-16T09:39:56.184384+0000","last_unstale":"2024-03-16T09:39:57.181798+0000","last_undegraded":"2024-03-16T09:39:57.181798+0000","last_fullsized":"2024-03-16T09:39:57.181798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:29.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269258+0000","last_change":"2024-03-16T09:39:56.184536+0000","last_active":"2024-03-16T09:39:57.269258+0000","last_peered":"2024-03-16T09:39:57.269258+0000","last_clean":"2024-03-16T09:39:57.269258+0000","last_became_active":"2024-03-16T09:39:56.184208+0000","last_became_peered":"2024-03-16T09:39:56.184208+0000","last_unstale":"2024-03-16T09:39:57.269258+0000","last_undegraded":"2024-03-16T09:39:57.269258+0000","last_fullsized":"2024-03-16T09:39:57.269258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:14.025625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.189057+0000","last_change":"2024-03-16T09:39:56.184331+0000","last_active":"2024-03-16T09:39:57.189057+0000","last_peered":"2024-03-16T09:39:57.189057+0000","last_clean":"2024-03-16T09:39:57.189057+0000","last_became_active":"2024-03-16T09:39:56.183919+0000","last_became_peered":"2024-03-16T09:39:56.183919+0000","last_unstale":"2024-03-16T09:39:57.189057+0000","last_undegraded":"2024-03-16T09:39:57.189057+0000","last_fullsized":"2024-03-16T09:39:57.189057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:59.692854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268567+0000","last_change":"2024-03-16T09:39:56.185607+0000","last_active":"2024-03-16T09:39:57.268567+0000","last_peered":"2024-03-16T09:39:57.268567+0000","last_clean":"2024-03-16T09:39:57.268567+0000","last_became_active":"2024-03-16T09:39:56.185278+0000","last_became_peered":"2024-03-16T09:39:56.185278+0000","last_unstale":"2024-03-16T09:39:57.268567+0000","last_undegraded":"2024-03-16T09:39:57.268567+0000","last_fullsized":"2024-03-16T09:39:57.268567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:20.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732837+0000","last_change":"2024-03-16T09:39:55.180852+0000","last_active":"2024-03-16T09:39:58.732837+0000","last_peered":"2024-03-16T09:39:58.732837+0000","last_clean":"2024-03-16T09:39:58.732837+0000","last_became_active":"2024-03-16T09:39:55.180496+0000","last_became_peered":"2024-03-16T09:39:55.180496+0000","last_unstale":"2024-03-16T09:39:58.732837+0000","last_undegraded":"2024-03-16T09:39:58.732837+0000","last_fullsized":"2024-03-16T09:39:58.732837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:15:28.548458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185400+0000","last_change":"2024-03-16T09:39:56.190110+0000","last_active":"2024-03-16T09:39:57.185400+0000","last_peered":"2024-03-16T09:39:57.185400+0000","last_clean":"2024-03-16T09:39:57.185400+0000","last_became_active":"2024-03-16T09:39:56.189711+0000","last_became_peered":"2024-03-16T09:39:56.189711+0000","last_unstale":"2024-03-16T09:39:57.185400+0000","last_undegraded":"2024-03-16T09:39:57.185400+0000","last_fullsized":"2024-03-16T09:39:57.185400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:17.353457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185703+0000","last_change":"2024-03-16T09:39:55.171142+0000","last_active":"2024-03-16T09:39:57.185703+0000","last_peered":"2024-03-16T09:39:57.185703+0000","last_clean":"2024-03-16T09:39:57.185703+0000","last_became_active":"2024-03-16T09:39:55.170793+0000","last_became_peered":"2024-03-16T09:39:55.170793+0000","last_unstale":"2024-03-16T09:39:57.185703+0000","last_undegraded":"2024-03-16T09:39:57.185703+0000","last_fullsized":"2024-03-16T09:39:57.185703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:36.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268798+0000","last_change":"2024-03-16T09:39:56.186802+0000","last_active":"2024-03-16T09:39:57.268798+0000","last_peered":"2024-03-16T09:39:57.268798+0000","last_clean":"2024-03-16T09:39:57.268798+0000","last_became_active":"2024-03-16T09:39:56.186398+0000","last_became_peered":"2024-03-16T09:39:56.186398+0000","last_unstale":"2024-03-16T09:39:57.268798+0000","last_undegraded":"2024-03-16T09:39:57.268798+0000","last_fullsized":"2024-03-16T09:39:57.268798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:05:56.209864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731812+0000","last_change":"2024-03-16T09:39:55.181894+0000","last_active":"2024-03-16T09:39:58.731812+0000","last_peered":"2024-03-16T09:39:58.731812+0000","last_clean":"2024-03-16T09:39:58.731812+0000","last_became_active":"2024-03-16T09:39:55.179434+0000","last_became_peered":"2024-03-16T09:39:55.179434+0000","last_unstale":"2024-03-16T09:39:58.731812+0000","last_undegraded":"2024-03-16T09:39:58.731812+0000","last_fullsized":"2024-03-16T09:39:58.731812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:16.153928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182169+0000","last_change":"2024-03-16T09:39:56.188464+0000","last_active":"2024-03-16T09:39:57.182169+0000","last_peered":"2024-03-16T09:39:57.182169+0000","last_clean":"2024-03-16T09:39:57.182169+0000","last_became_active":"2024-03-16T09:39:56.188125+0000","last_became_peered":"2024-03-16T09:39:56.188125+0000","last_unstale":"2024-03-16T09:39:57.182169+0000","last_undegraded":"2024-03-16T09:39:57.182169+0000","last_fullsized":"2024-03-16T09:39:57.182169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:46:49.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191478+0000","last_change":"2024-03-16T09:39:55.179379+0000","last_active":"2024-03-16T09:39:57.191478+0000","last_peered":"2024-03-16T09:39:57.191478+0000","last_clean":"2024-03-16T09:39:57.191478+0000","last_became_active":"2024-03-16T09:39:55.178755+0000","last_became_peered":"2024-03-16T09:39:55.178755+0000","last_unstale":"2024-03-16T09:39:57.191478+0000","last_undegraded":"2024-03-16T09:39:57.191478+0000","last_fullsized":"2024-03-16T09:39:57.191478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:07.131417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.197489+0000","last_change":"2024-03-16T09:39:56.196655+0000","last_active":"2024-03-16T09:39:56.197489+0000","last_peered":"2024-03-16T09:39:56.197489+0000","last_clean":"2024-03-16T09:39:56.197489+0000","last_became_active":"2024-03-16T09:39:56.196344+0000","last_became_peered":"2024-03-16T09:39:56.196344+0000","last_unstale":"2024-03-16T09:39:56.197489+0000","last_undegraded":"2024-03-16T09:39:56.197489+0000","last_fullsized":"2024-03-16T09:39:56.197489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:15.437849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.194714+0000","last_change":"2024-03-16T09:39:55.182012+0000","last_active":"2024-03-16T09:39:56.194714+0000","last_peered":"2024-03-16T09:39:56.194714+0000","last_clean":"2024-03-16T09:39:56.194714+0000","last_became_active":"2024-03-16T09:39:55.178651+0000","last_became_peered":"2024-03-16T09:39:55.178651+0000","last_unstale":"2024-03-16T09:39:56.194714+0000","last_undegraded":"2024-03-16T09:39:56.194714+0000","last_fullsized":"2024-03-16T09:39:56.194714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:50:19.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191077+0000","last_change":"2024-03-16T09:39:56.179645+0000","last_active":"2024-03-16T09:39:57.191077+0000","last_peered":"2024-03-16T09:39:57.191077+0000","last_clean":"2024-03-16T09:39:57.191077+0000","last_became_active":"2024-03-16T09:39:56.179257+0000","last_became_peered":"2024-03-16T09:39:56.179257+0000","last_unstale":"2024-03-16T09:39:57.191077+0000","last_undegraded":"2024-03-16T09:39:57.191077+0000","last_fullsized":"2024-03-16T09:39:57.191077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:30:09.104630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731495+0000","last_change":"2024-03-16T09:39:55.178257+0000","last_active":"2024-03-16T09:39:58.731495+0000","last_peered":"2024-03-16T09:39:58.731495+0000","last_clean":"2024-03-16T09:39:58.731495+0000","last_became_active":"2024-03-16T09:39:55.177984+0000","last_became_peered":"2024-03-16T09:39:55.177984+0000","last_unstale":"2024-03-16T09:39:58.731495+0000","last_undegraded":"2024-03-16T09:39:58.731495+0000","last_fullsized":"2024-03-16T09:39:58.731495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.385624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.198862+0000","last_change":"2024-03-16T09:39:56.196821+0000","last_active":"2024-03-16T09:39:56.198862+0000","last_peered":"2024-03-16T09:39:56.198862+0000","last_clean":"2024-03-16T09:39:56.198862+0000","last_became_active":"2024-03-16T09:39:56.196522+0000","last_became_peered":"2024-03-16T09:39:56.196522+0000","last_unstale":"2024-03-16T09:39:56.198862+0000","last_undegraded":"2024-03-16T09:39:56.198862+0000","last_fullsized":"2024-03-16T09:39:56.198862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:19:42.587126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732000+0000","last_change":"2024-03-16T09:39:55.170290+0000","last_active":"2024-03-16T09:39:58.732000+0000","last_peered":"2024-03-16T09:39:58.732000+0000","last_clean":"2024-03-16T09:39:58.732000+0000","last_became_active":"2024-03-16T09:39:55.169956+0000","last_became_peered":"2024-03-16T09:39:55.169956+0000","last_unstale":"2024-03-16T09:39:58.732000+0000","last_undegraded":"2024-03-16T09:39:58.732000+0000","last_fullsized":"2024-03-16T09:39:58.732000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:01:31.606917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185671+0000","last_change":"2024-03-16T09:39:56.189155+0000","last_active":"2024-03-16T09:39:57.185671+0000","last_peered":"2024-03-16T09:39:57.185671+0000","last_clean":"2024-03-16T09:39:57.185671+0000","last_became_active":"2024-03-16T09:39:56.188827+0000","last_became_peered":"2024-03-16T09:39:56.188827+0000","last_unstale":"2024-03-16T09:39:57.185671+0000","last_undegraded":"2024-03-16T09:39:57.185671+0000","last_fullsized":"2024-03-16T09:39:57.185671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:54.370754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.172364+0000","last_change":"2024-03-16T09:39:55.181886+0000","last_active":"2024-03-16T09:39:56.172364+0000","last_peered":"2024-03-16T09:39:56.172364+0000","last_clean":"2024-03-16T09:39:56.172364+0000","last_became_active":"2024-03-16T09:39:55.179836+0000","last_became_peered":"2024-03-16T09:39:55.179836+0000","last_unstale":"2024-03-16T09:39:56.172364+0000","last_undegraded":"2024-03-16T09:39:56.172364+0000","last_fullsized":"2024-03-16T09:39:56.172364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:52:38.772861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.188909+0000","last_change":"2024-03-16T09:39:56.185892+0000","last_active":"2024-03-16T09:39:57.188909+0000","last_peered":"2024-03-16T09:39:57.188909+0000","last_clean":"2024-03-16T09:39:57.188909+0000","last_became_active":"2024-03-16T09:39:56.185459+0000","last_became_peered":"2024-03-16T09:39:56.185459+0000","last_unstale":"2024-03-16T09:39:57.188909+0000","last_undegraded":"2024-03-16T09:39:57.188909+0000","last_fullsized":"2024-03-16T09:39:57.188909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:10.421054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713808+0000","last_change":"2024-03-16T09:39:55.177607+0000","last_active":"2024-03-16T09:39:57.713808+0000","last_peered":"2024-03-16T09:39:57.713808+0000","last_clean":"2024-03-16T09:39:57.713808+0000","last_became_active":"2024-03-16T09:39:55.177265+0000","last_became_peered":"2024-03-16T09:39:55.177265+0000","last_unstale":"2024-03-16T09:39:57.713808+0000","last_undegraded":"2024-03-16T09:39:57.713808+0000","last_fullsized":"2024-03-16T09:39:57.713808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:47.849041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182281+0000","last_change":"2024-03-16T09:39:56.189172+0000","last_active":"2024-03-16T09:39:57.182281+0000","last_peered":"2024-03-16T09:39:57.182281+0000","last_clean":"2024-03-16T09:39:57.182281+0000","last_became_active":"2024-03-16T09:39:56.188798+0000","last_became_peered":"2024-03-16T09:39:56.188798+0000","last_unstale":"2024-03-16T09:39:57.182281+0000","last_undegraded":"2024-03-16T09:39:57.182281+0000","last_fullsized":"2024-03-16T09:39:57.182281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:02:11.119293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.712618+0000","last_change":"2024-03-16T09:39:55.180369+0000","last_active":"2024-03-16T09:39:57.712618+0000","last_peered":"2024-03-16T09:39:57.712618+0000","last_clean":"2024-03-16T09:39:57.712618+0000","last_became_active":"2024-03-16T09:39:55.180053+0000","last_became_peered":"2024-03-16T09:39:55.180053+0000","last_unstale":"2024-03-16T09:39:57.712618+0000","last_undegraded":"2024-03-16T09:39:57.712618+0000","last_fullsized":"2024-03-16T09:39:57.712618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:44.990028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268679+0000","last_change":"2024-03-16T09:39:56.198454+0000","last_active":"2024-03-16T09:39:57.268679+0000","last_peered":"2024-03-16T09:39:57.268679+0000","last_clean":"2024-03-16T09:39:57.268679+0000","last_became_active":"2024-03-16T09:39:56.198097+0000","last_became_peered":"2024-03-16T09:39:56.198097+0000","last_unstale":"2024-03-16T09:39:57.268679+0000","last_undegraded":"2024-03-16T09:39:57.268679+0000","last_fullsized":"2024-03-16T09:39:57.268679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:12.494363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.195938+0000","last_change":"2024-03-16T09:39:55.174221+0000","last_active":"2024-03-16T09:39:56.195938+0000","last_peered":"2024-03-16T09:39:56.195938+0000","last_clean":"2024-03-16T09:39:56.195938+0000","last_became_active":"2024-03-16T09:39:55.173920+0000","last_became_peered":"2024-03-16T09:39:55.173920+0000","last_unstale":"2024-03-16T09:39:56.195938+0000","last_undegraded":"2024-03-16T09:39:56.195938+0000","last_fullsized":"2024-03-16T09:39:56.195938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:25.108414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.182119+0000","last_change":"2024-03-16T09:39:56.196849+0000","last_active":"2024-03-16T09:39:57.182119+0000","last_peered":"2024-03-16T09:39:57.182119+0000","last_clean":"2024-03-16T09:39:57.182119+0000","last_became_active":"2024-03-16T09:39:56.196515+0000","last_became_peered":"2024-03-16T09:39:56.196515+0000","last_unstale":"2024-03-16T09:39:57.182119+0000","last_undegraded":"2024-03-16T09:39:57.182119+0000","last_fullsized":"2024-03-16T09:39:57.182119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:03.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:40:00.065303+0000","last_change":"2024-03-16T09:39:55.177788+0000","last_active":"2024-03-16T09:40:00.065303+0000","last_peered":"2024-03-16T09:40:00.065303+0000","last_clean":"2024-03-16T09:40:00.065303+0000","last_became_active":"2024-03-16T09:39:55.177443+0000","last_became_peered":"2024-03-16T09:39:55.177443+0000","last_unstale":"2024-03-16T09:40:00.065303+0000","last_undegraded":"2024-03-16T09:40:00.065303+0000","last_fullsized":"2024-03-16T09:40:00.065303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:39.221712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4018,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191241+0000","last_change":"2024-03-16T09:39:56.178976+0000","last_active":"2024-03-16T09:39:57.191241+0000","last_peered":"2024-03-16T09:39:57.191241+0000","last_clean":"2024-03-16T09:39:57.191241+0000","last_became_active":"2024-03-16T09:39:56.178455+0000","last_became_peered":"2024-03-16T09:39:56.178455+0000","last_unstale":"2024-03-16T09:39:57.191241+0000","last_undegraded":"2024-03-16T09:39:57.191241+0000","last_fullsized":"2024-03-16T09:39:57.191241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:52:08.064792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731610+0000","last_change":"2024-03-16T09:39:55.182579+0000","last_active":"2024-03-16T09:39:58.731610+0000","last_peered":"2024-03-16T09:39:58.731610+0000","last_clean":"2024-03-16T09:39:58.731610+0000","last_became_active":"2024-03-16T09:39:55.181298+0000","last_became_peered":"2024-03-16T09:39:55.181298+0000","last_unstale":"2024-03-16T09:39:58.731610+0000","last_undegraded":"2024-03-16T09:39:58.731610+0000","last_fullsized":"2024-03-16T09:39:58.731610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:35.768812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268361+0000","last_change":"2024-03-16T09:39:56.188519+0000","last_active":"2024-03-16T09:39:57.268361+0000","last_peered":"2024-03-16T09:39:57.268361+0000","last_clean":"2024-03-16T09:39:57.268361+0000","last_became_active":"2024-03-16T09:39:56.188031+0000","last_became_peered":"2024-03-16T09:39:56.188031+0000","last_unstale":"2024-03-16T09:39:57.268361+0000","last_undegraded":"2024-03-16T09:39:57.268361+0000","last_fullsized":"2024-03-16T09:39:57.268361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:20:31.765657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.172284+0000","last_change":"2024-03-16T09:39:55.177523+0000","last_active":"2024-03-16T09:39:56.172284+0000","last_peered":"2024-03-16T09:39:56.172284+0000","last_clean":"2024-03-16T09:39:56.172284+0000","last_became_active":"2024-03-16T09:39:55.177143+0000","last_became_peered":"2024-03-16T09:39:55.177143+0000","last_unstale":"2024-03-16T09:39:56.172284+0000","last_undegraded":"2024-03-16T09:39:56.172284+0000","last_fullsized":"2024-03-16T09:39:56.172284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:05.357611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269548+0000","last_change":"2024-03-16T09:39:56.196794+0000","last_active":"2024-03-16T09:39:57.269548+0000","last_peered":"2024-03-16T09:39:57.269548+0000","last_clean":"2024-03-16T09:39:57.269548+0000","last_became_active":"2024-03-16T09:39:56.196425+0000","last_became_peered":"2024-03-16T09:39:56.196425+0000","last_unstale":"2024-03-16T09:39:57.269548+0000","last_undegraded":"2024-03-16T09:39:57.269548+0000","last_fullsized":"2024-03-16T09:39:57.269548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:26:45.952779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.735522+0000","last_change":"2024-03-16T09:39:55.175419+0000","last_active":"2024-03-16T09:39:58.735522+0000","last_peered":"2024-03-16T09:39:58.735522+0000","last_clean":"2024-03-16T09:39:58.735522+0000","last_became_active":"2024-03-16T09:39:55.175126+0000","last_became_peered":"2024-03-16T09:39:55.175126+0000","last_unstale":"2024-03-16T09:39:58.735522+0000","last_undegraded":"2024-03-16T09:39:58.735522+0000","last_fullsized":"2024-03-16T09:39:58.735522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:11.696911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269215+0000","last_change":"2024-03-16T09:39:56.176714+0000","last_active":"2024-03-16T09:39:57.269215+0000","last_peered":"2024-03-16T09:39:57.269215+0000","last_clean":"2024-03-16T09:39:57.269215+0000","last_became_active":"2024-03-16T09:39:56.176245+0000","last_became_peered":"2024-03-16T09:39:56.176245+0000","last_unstale":"2024-03-16T09:39:57.269215+0000","last_undegraded":"2024-03-16T09:39:57.269215+0000","last_fullsized":"2024-03-16T09:39:57.269215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:32.430207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.733635+0000","last_change":"2024-03-16T09:39:55.181486+0000","last_active":"2024-03-16T09:39:58.733635+0000","last_peered":"2024-03-16T09:39:58.733635+0000","last_clean":"2024-03-16T09:39:58.733635+0000","last_became_active":"2024-03-16T09:39:55.180128+0000","last_became_peered":"2024-03-16T09:39:55.180128+0000","last_unstale":"2024-03-16T09:39:58.733635+0000","last_undegraded":"2024-03-16T09:39:58.733635+0000","last_fullsized":"2024-03-16T09:39:58.733635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:45.323870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.199006+0000","last_change":"2024-03-16T09:39:56.196692+0000","last_active":"2024-03-16T09:39:56.199006+0000","last_peered":"2024-03-16T09:39:56.199006+0000","last_clean":"2024-03-16T09:39:56.199006+0000","last_became_active":"2024-03-16T09:39:56.196403+0000","last_became_peered":"2024-03-16T09:39:56.196403+0000","last_unstale":"2024-03-16T09:39:56.199006+0000","last_undegraded":"2024-03-16T09:39:56.199006+0000","last_fullsized":"2024-03-16T09:39:56.199006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:16.953824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:59.728978+0000","last_change":"2024-03-16T09:39:55.184156+0000","last_active":"2024-03-16T09:39:59.728978+0000","last_peered":"2024-03-16T09:39:59.728978+0000","last_clean":"2024-03-16T09:39:59.728978+0000","last_became_active":"2024-03-16T09:39:55.183563+0000","last_became_peered":"2024-03-16T09:39:55.183563+0000","last_unstale":"2024-03-16T09:39:59.728978+0000","last_undegraded":"2024-03-16T09:39:59.728978+0000","last_fullsized":"2024-03-16T09:39:59.728978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:22.376434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269043+0000","last_change":"2024-03-16T09:39:56.196699+0000","last_active":"2024-03-16T09:39:57.269043+0000","last_peered":"2024-03-16T09:39:57.269043+0000","last_clean":"2024-03-16T09:39:57.269043+0000","last_became_active":"2024-03-16T09:39:56.196386+0000","last_became_peered":"2024-03-16T09:39:56.196386+0000","last_unstale":"2024-03-16T09:39:57.269043+0000","last_undegraded":"2024-03-16T09:39:57.269043+0000","last_fullsized":"2024-03-16T09:39:57.269043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:20:32.734352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.188964+0000","last_change":"2024-03-16T09:39:55.175666+0000","last_active":"2024-03-16T09:39:57.188964+0000","last_peered":"2024-03-16T09:39:57.188964+0000","last_clean":"2024-03-16T09:39:57.188964+0000","last_became_active":"2024-03-16T09:39:55.175337+0000","last_became_peered":"2024-03-16T09:39:55.175337+0000","last_unstale":"2024-03-16T09:39:57.188964+0000","last_undegraded":"2024-03-16T09:39:57.188964+0000","last_fullsized":"2024-03-16T09:39:57.188964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:45:13.935669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268876+0000","last_change":"2024-03-16T09:39:56.184878+0000","last_active":"2024-03-16T09:39:57.268876+0000","last_peered":"2024-03-16T09:39:57.268876+0000","last_clean":"2024-03-16T09:39:57.268876+0000","last_became_active":"2024-03-16T09:39:56.184480+0000","last_became_peered":"2024-03-16T09:39:56.184480+0000","last_unstale":"2024-03-16T09:39:57.268876+0000","last_undegraded":"2024-03-16T09:39:57.268876+0000","last_fullsized":"2024-03-16T09:39:57.268876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:40:01.286993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269514+0000","last_change":"2024-03-16T09:39:55.179189+0000","last_active":"2024-03-16T09:39:57.269514+0000","last_peered":"2024-03-16T09:39:57.269514+0000","last_clean":"2024-03-16T09:39:57.269514+0000","last_became_active":"2024-03-16T09:39:55.178859+0000","last_became_peered":"2024-03-16T09:39:55.178859+0000","last_unstale":"2024-03-16T09:39:57.269514+0000","last_undegraded":"2024-03-16T09:39:57.269514+0000","last_fullsized":"2024-03-16T09:39:57.269514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:54:15.498386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269147+0000","last_change":"2024-03-16T09:39:56.200665+0000","last_active":"2024-03-16T09:39:57.269147+0000","last_peered":"2024-03-16T09:39:57.269147+0000","last_clean":"2024-03-16T09:39:57.269147+0000","last_became_active":"2024-03-16T09:39:56.200326+0000","last_became_peered":"2024-03-16T09:39:56.200326+0000","last_unstale":"2024-03-16T09:39:57.269147+0000","last_undegraded":"2024-03-16T09:39:57.269147+0000","last_fullsized":"2024-03-16T09:39:57.269147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:34.778630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.184330+0000","last_change":"2024-03-16T09:39:56.182670+0000","last_active":"2024-03-16T09:39:56.184330+0000","last_peered":"2024-03-16T09:39:56.184330+0000","last_clean":"2024-03-16T09:39:56.184330+0000","last_became_active":"2024-03-16T09:39:56.182009+0000","last_became_peered":"2024-03-16T09:39:56.182009+0000","last_unstale":"2024-03-16T09:39:56.184330+0000","last_undegraded":"2024-03-16T09:39:56.184330+0000","last_fullsized":"2024-03-16T09:39:56.184330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:02:32.399218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:59.735655+0000","last_change":"2024-03-16T09:39:55.171609+0000","last_active":"2024-03-16T09:39:59.735655+0000","last_peered":"2024-03-16T09:39:59.735655+0000","last_clean":"2024-03-16T09:39:59.735655+0000","last_became_active":"2024-03-16T09:39:55.171301+0000","last_became_peered":"2024-03-16T09:39:55.171301+0000","last_unstale":"2024-03-16T09:39:59.735655+0000","last_undegraded":"2024-03-16T09:39:59.735655+0000","last_fullsized":"2024-03-16T09:39:59.735655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:28.265342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185679+0000","last_change":"2024-03-16T09:39:56.190270+0000","last_active":"2024-03-16T09:39:57.185679+0000","last_peered":"2024-03-16T09:39:57.185679+0000","last_clean":"2024-03-16T09:39:57.185679+0000","last_became_active":"2024-03-16T09:39:56.189882+0000","last_became_peered":"2024-03-16T09:39:56.189882+0000","last_unstale":"2024-03-16T09:39:57.185679+0000","last_undegraded":"2024-03-16T09:39:57.185679+0000","last_fullsized":"2024-03-16T09:39:57.185679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:47.290540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.731782+0000","last_change":"2024-03-16T09:39:55.172824+0000","last_active":"2024-03-16T09:39:58.731782+0000","last_peered":"2024-03-16T09:39:58.731782+0000","last_clean":"2024-03-16T09:39:58.731782+0000","last_became_active":"2024-03-16T09:39:55.172493+0000","last_became_peered":"2024-03-16T09:39:55.172493+0000","last_unstale":"2024-03-16T09:39:58.731782+0000","last_undegraded":"2024-03-16T09:39:58.731782+0000","last_fullsized":"2024-03-16T09:39:58.731782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:39.188176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269322+0000","last_change":"2024-03-16T09:38:43.693644+0000","last_active":"2024-03-16T09:39:57.269322+0000","last_peered":"2024-03-16T09:39:57.269322+0000","last_clean":"2024-03-16T09:39:57.269322+0000","last_became_active":"2024-03-16T09:38:43.693202+0000","last_became_peered":"2024-03-16T09:38:43.693202+0000","last_unstale":"2024-03-16T09:39:57.269322+0000","last_undegraded":"2024-03-16T09:39:57.269322+0000","last_fullsized":"2024-03-16T09:39:57.269322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_clean_scrub_stamp":"2024-03-16T09:35:24.153508+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:43:06.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268148+0000","last_change":"2024-03-16T09:39:56.197109+0000","last_active":"2024-03-16T09:39:57.268148+0000","last_peered":"2024-03-16T09:39:57.268148+0000","last_clean":"2024-03-16T09:39:57.268148+0000","last_became_active":"2024-03-16T09:39:56.196775+0000","last_became_peered":"2024-03-16T09:39:56.196775+0000","last_unstale":"2024-03-16T09:39:57.268148+0000","last_undegraded":"2024-03-16T09:39:57.268148+0000","last_fullsized":"2024-03-16T09:39:57.268148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:29:10.401423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732143+0000","last_change":"2024-03-16T09:39:55.175744+0000","last_active":"2024-03-16T09:39:58.732143+0000","last_peered":"2024-03-16T09:39:58.732143+0000","last_clean":"2024-03-16T09:39:58.732143+0000","last_became_active":"2024-03-16T09:39:55.175445+0000","last_became_peered":"2024-03-16T09:39:55.175445+0000","last_unstale":"2024-03-16T09:39:58.732143+0000","last_undegraded":"2024-03-16T09:39:58.732143+0000","last_fullsized":"2024-03-16T09:39:58.732143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:39.494340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713098+0000","last_change":"2024-03-16T09:39:55.183441+0000","last_active":"2024-03-16T09:39:57.713098+0000","last_peered":"2024-03-16T09:39:57.713098+0000","last_clean":"2024-03-16T09:39:57.713098+0000","last_became_active":"2024-03-16T09:39:55.183115+0000","last_became_peered":"2024-03-16T09:39:55.183115+0000","last_unstale":"2024-03-16T09:39:57.713098+0000","last_undegraded":"2024-03-16T09:39:57.713098+0000","last_fullsized":"2024-03-16T09:39:57.713098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:55:31.525201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268618+0000","last_change":"2024-03-16T09:39:56.179205+0000","last_active":"2024-03-16T09:39:57.268618+0000","last_peered":"2024-03-16T09:39:57.268618+0000","last_clean":"2024-03-16T09:39:57.268618+0000","last_became_active":"2024-03-16T09:39:56.178849+0000","last_became_peered":"2024-03-16T09:39:56.178849+0000","last_unstale":"2024-03-16T09:39:57.268618+0000","last_undegraded":"2024-03-16T09:39:57.268618+0000","last_fullsized":"2024-03-16T09:39:57.268618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:10.016099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:40:00.188079+0000","last_change":"2024-03-16T09:39:55.180859+0000","last_active":"2024-03-16T09:40:00.188079+0000","last_peered":"2024-03-16T09:40:00.188079+0000","last_clean":"2024-03-16T09:40:00.188079+0000","last_became_active":"2024-03-16T09:39:55.180514+0000","last_became_peered":"2024-03-16T09:39:55.180514+0000","last_unstale":"2024-03-16T09:40:00.188079+0000","last_undegraded":"2024-03-16T09:40:00.188079+0000","last_fullsized":"2024-03-16T09:40:00.188079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:46.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185799+0000","last_change":"2024-03-16T09:39:56.187626+0000","last_active":"2024-03-16T09:39:57.185799+0000","last_peered":"2024-03-16T09:39:57.185799+0000","last_clean":"2024-03-16T09:39:57.185799+0000","last_became_active":"2024-03-16T09:39:56.187284+0000","last_became_peered":"2024-03-16T09:39:56.187284+0000","last_unstale":"2024-03-16T09:39:57.185799+0000","last_undegraded":"2024-03-16T09:39:57.185799+0000","last_fullsized":"2024-03-16T09:39:57.185799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:23.511603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713140+0000","last_change":"2024-03-16T09:39:55.172912+0000","last_active":"2024-03-16T09:39:57.713140+0000","last_peered":"2024-03-16T09:39:57.713140+0000","last_clean":"2024-03-16T09:39:57.713140+0000","last_became_active":"2024-03-16T09:39:55.172592+0000","last_became_peered":"2024-03-16T09:39:55.172592+0000","last_unstale":"2024-03-16T09:39:57.713140+0000","last_undegraded":"2024-03-16T09:39:57.713140+0000","last_fullsized":"2024-03-16T09:39:57.713140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:24.344588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268563+0000","last_change":"2024-03-16T09:39:56.197652+0000","last_active":"2024-03-16T09:39:57.268563+0000","last_peered":"2024-03-16T09:39:57.268563+0000","last_clean":"2024-03-16T09:39:57.268563+0000","last_became_active":"2024-03-16T09:39:56.197230+0000","last_became_peered":"2024-03-16T09:39:56.197230+0000","last_unstale":"2024-03-16T09:39:57.268563+0000","last_undegraded":"2024-03-16T09:39:57.268563+0000","last_fullsized":"2024-03-16T09:39:57.268563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:45:08.661372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.715034+0000","last_change":"2024-03-16T09:39:55.175140+0000","last_active":"2024-03-16T09:39:57.715034+0000","last_peered":"2024-03-16T09:39:57.715034+0000","last_clean":"2024-03-16T09:39:57.715034+0000","last_became_active":"2024-03-16T09:39:55.174764+0000","last_became_peered":"2024-03-16T09:39:55.174764+0000","last_unstale":"2024-03-16T09:39:57.715034+0000","last_undegraded":"2024-03-16T09:39:57.715034+0000","last_fullsized":"2024-03-16T09:39:57.715034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:03:04.591466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.191248+0000","last_change":"2024-03-16T09:39:56.179326+0000","last_active":"2024-03-16T09:39:57.191248+0000","last_peered":"2024-03-16T09:39:57.191248+0000","last_clean":"2024-03-16T09:39:57.191248+0000","last_became_active":"2024-03-16T09:39:56.178681+0000","last_became_peered":"2024-03-16T09:39:56.178681+0000","last_unstale":"2024-03-16T09:39:57.191248+0000","last_undegraded":"2024-03-16T09:39:57.191248+0000","last_fullsized":"2024-03-16T09:39:57.191248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:26.137807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.713555+0000","last_change":"2024-03-16T09:39:55.173820+0000","last_active":"2024-03-16T09:39:57.713555+0000","last_peered":"2024-03-16T09:39:57.713555+0000","last_clean":"2024-03-16T09:39:57.713555+0000","last_became_active":"2024-03-16T09:39:55.173472+0000","last_became_peered":"2024-03-16T09:39:55.173472+0000","last_unstale":"2024-03-16T09:39:57.713555+0000","last_undegraded":"2024-03-16T09:39:57.713555+0000","last_fullsized":"2024-03-16T09:39:57.713555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:14.082367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268901+0000","last_change":"2024-03-16T09:39:56.178535+0000","last_active":"2024-03-16T09:39:57.268901+0000","last_peered":"2024-03-16T09:39:57.268901+0000","last_clean":"2024-03-16T09:39:57.268901+0000","last_became_active":"2024-03-16T09:39:56.178210+0000","last_became_peered":"2024-03-16T09:39:56.178210+0000","last_unstale":"2024-03-16T09:39:57.268901+0000","last_undegraded":"2024-03-16T09:39:57.268901+0000","last_fullsized":"2024-03-16T09:39:57.268901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:18.680469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268109+0000","last_change":"2024-03-16T09:39:55.183074+0000","last_active":"2024-03-16T09:39:57.268109+0000","last_peered":"2024-03-16T09:39:57.268109+0000","last_clean":"2024-03-16T09:39:57.268109+0000","last_became_active":"2024-03-16T09:39:55.182695+0000","last_became_peered":"2024-03-16T09:39:55.182695+0000","last_unstale":"2024-03-16T09:39:57.268109+0000","last_undegraded":"2024-03-16T09:39:57.268109+0000","last_fullsized":"2024-03-16T09:39:57.268109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:51.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185745+0000","last_change":"2024-03-16T09:39:56.193434+0000","last_active":"2024-03-16T09:39:57.185745+0000","last_peered":"2024-03-16T09:39:57.185745+0000","last_clean":"2024-03-16T09:39:57.185745+0000","last_became_active":"2024-03-16T09:39:56.193051+0000","last_became_peered":"2024-03-16T09:39:56.193051+0000","last_unstale":"2024-03-16T09:39:57.185745+0000","last_undegraded":"2024-03-16T09:39:57.185745+0000","last_fullsized":"2024-03-16T09:39:57.185745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:12:26.767117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.712591+0000","last_change":"2024-03-16T09:39:55.179916+0000","last_active":"2024-03-16T09:39:57.712591+0000","last_peered":"2024-03-16T09:39:57.712591+0000","last_clean":"2024-03-16T09:39:57.712591+0000","last_became_active":"2024-03-16T09:39:55.179635+0000","last_became_peered":"2024-03-16T09:39:55.179635+0000","last_unstale":"2024-03-16T09:39:57.712591+0000","last_undegraded":"2024-03-16T09:39:57.712591+0000","last_fullsized":"2024-03-16T09:39:57.712591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:08.955039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269687+0000","last_change":"2024-03-16T09:39:56.194044+0000","last_active":"2024-03-16T09:39:57.269687+0000","last_peered":"2024-03-16T09:39:57.269687+0000","last_clean":"2024-03-16T09:39:57.269687+0000","last_became_active":"2024-03-16T09:39:56.193490+0000","last_became_peered":"2024-03-16T09:39:56.193490+0000","last_unstale":"2024-03-16T09:39:57.269687+0000","last_undegraded":"2024-03-16T09:39:57.269687+0000","last_fullsized":"2024-03-16T09:39:57.269687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:19:59.916892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:58.732272+0000","last_change":"2024-03-16T09:39:55.182652+0000","last_active":"2024-03-16T09:39:58.732272+0000","last_peered":"2024-03-16T09:39:58.732272+0000","last_clean":"2024-03-16T09:39:58.732272+0000","last_became_active":"2024-03-16T09:39:55.181432+0000","last_became_peered":"2024-03-16T09:39:55.181432+0000","last_unstale":"2024-03-16T09:39:58.732272+0000","last_undegraded":"2024-03-16T09:39:58.732272+0000","last_fullsized":"2024-03-16T09:39:58.732272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:34.959845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.270027+0000","last_change":"2024-03-16T09:39:56.188970+0000","last_active":"2024-03-16T09:39:57.270027+0000","last_peered":"2024-03-16T09:39:57.270027+0000","last_clean":"2024-03-16T09:39:57.270027+0000","last_became_active":"2024-03-16T09:39:56.188607+0000","last_became_peered":"2024-03-16T09:39:56.188607+0000","last_unstale":"2024-03-16T09:39:57.270027+0000","last_undegraded":"2024-03-16T09:39:57.270027+0000","last_fullsized":"2024-03-16T09:39:57.270027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:39:24.532114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.195943+0000","last_change":"2024-03-16T09:39:55.181908+0000","last_active":"2024-03-16T09:39:56.195943+0000","last_peered":"2024-03-16T09:39:56.195943+0000","last_clean":"2024-03-16T09:39:56.195943+0000","last_became_active":"2024-03-16T09:39:55.181055+0000","last_became_peered":"2024-03-16T09:39:55.181055+0000","last_unstale":"2024-03-16T09:39:56.195943+0000","last_undegraded":"2024-03-16T09:39:56.195943+0000","last_fullsized":"2024-03-16T09:39:56.195943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:38.256664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269617+0000","last_change":"2024-03-16T09:39:56.189743+0000","last_active":"2024-03-16T09:39:57.269617+0000","last_peered":"2024-03-16T09:39:57.269617+0000","last_clean":"2024-03-16T09:39:57.269617+0000","last_became_active":"2024-03-16T09:39:56.189339+0000","last_became_peered":"2024-03-16T09:39:56.189339+0000","last_unstale":"2024-03-16T09:39:57.269617+0000","last_undegraded":"2024-03-16T09:39:57.269617+0000","last_fullsized":"2024-03-16T09:39:57.269617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:54.694191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268125+0000","last_change":"2024-03-16T09:39:55.182357+0000","last_active":"2024-03-16T09:39:57.268125+0000","last_peered":"2024-03-16T09:39:57.268125+0000","last_clean":"2024-03-16T09:39:57.268125+0000","last_became_active":"2024-03-16T09:39:55.182053+0000","last_became_peered":"2024-03-16T09:39:55.182053+0000","last_unstale":"2024-03-16T09:39:57.268125+0000","last_undegraded":"2024-03-16T09:39:57.268125+0000","last_fullsized":"2024-03-16T09:39:57.268125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:40.735596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.269683+0000","last_change":"2024-03-16T09:39:56.181636+0000","last_active":"2024-03-16T09:39:57.269683+0000","last_peered":"2024-03-16T09:39:57.269683+0000","last_clean":"2024-03-16T09:39:57.269683+0000","last_became_active":"2024-03-16T09:39:56.181309+0000","last_became_peered":"2024-03-16T09:39:56.181309+0000","last_unstale":"2024-03-16T09:39:57.269683+0000","last_undegraded":"2024-03-16T09:39:57.269683+0000","last_fullsized":"2024-03-16T09:39:57.269683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:36.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268746+0000","last_change":"2024-03-16T09:39:56.186640+0000","last_active":"2024-03-16T09:39:57.268746+0000","last_peered":"2024-03-16T09:39:57.268746+0000","last_clean":"2024-03-16T09:39:57.268746+0000","last_became_active":"2024-03-16T09:39:56.186311+0000","last_became_peered":"2024-03-16T09:39:56.186311+0000","last_unstale":"2024-03-16T09:39:57.268746+0000","last_undegraded":"2024-03-16T09:39:57.268746+0000","last_fullsized":"2024-03-16T09:39:57.268746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:58:44.451068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268563+0000","last_change":"2024-03-16T09:39:56.196887+0000","last_active":"2024-03-16T09:39:57.268563+0000","last_peered":"2024-03-16T09:39:57.268563+0000","last_clean":"2024-03-16T09:39:57.268563+0000","last_became_active":"2024-03-16T09:39:56.196568+0000","last_became_peered":"2024-03-16T09:39:56.196568+0000","last_unstale":"2024-03-16T09:39:57.268563+0000","last_undegraded":"2024-03-16T09:39:57.268563+0000","last_fullsized":"2024-03-16T09:39:57.268563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:08:35.577630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268213+0000","last_change":"2024-03-16T09:39:56.186394+0000","last_active":"2024-03-16T09:39:57.268213+0000","last_peered":"2024-03-16T09:39:57.268213+0000","last_clean":"2024-03-16T09:39:57.268213+0000","last_became_active":"2024-03-16T09:39:56.186053+0000","last_became_peered":"2024-03-16T09:39:56.186053+0000","last_unstale":"2024-03-16T09:39:57.268213+0000","last_undegraded":"2024-03-16T09:39:57.268213+0000","last_fullsized":"2024-03-16T09:39:57.268213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:41.529601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268396+0000","last_change":"2024-03-16T09:39:56.197622+0000","last_active":"2024-03-16T09:39:57.268396+0000","last_peered":"2024-03-16T09:39:57.268396+0000","last_clean":"2024-03-16T09:39:57.268396+0000","last_became_active":"2024-03-16T09:39:56.196885+0000","last_became_peered":"2024-03-16T09:39:56.196885+0000","last_unstale":"2024-03-16T09:39:57.268396+0000","last_undegraded":"2024-03-16T09:39:57.268396+0000","last_fullsized":"2024-03-16T09:39:57.268396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:08:50.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268461+0000","last_change":"2024-03-16T09:39:56.192318+0000","last_active":"2024-03-16T09:39:57.268461+0000","last_peered":"2024-03-16T09:39:57.268461+0000","last_clean":"2024-03-16T09:39:57.268461+0000","last_became_active":"2024-03-16T09:39:56.191998+0000","last_became_peered":"2024-03-16T09:39:56.191998+0000","last_unstale":"2024-03-16T09:39:57.268461+0000","last_undegraded":"2024-03-16T09:39:57.268461+0000","last_fullsized":"2024-03-16T09:39:57.268461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:51:47.025415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268942+0000","last_change":"2024-03-16T09:39:56.181262+0000","last_active":"2024-03-16T09:39:57.268942+0000","last_peered":"2024-03-16T09:39:57.268942+0000","last_clean":"2024-03-16T09:39:57.268942+0000","last_became_active":"2024-03-16T09:39:56.180916+0000","last_became_peered":"2024-03-16T09:39:56.180916+0000","last_unstale":"2024-03-16T09:39:57.268942+0000","last_undegraded":"2024-03-16T09:39:57.268942+0000","last_fullsized":"2024-03-16T09:39:57.268942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:34:45.056025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268834+0000","last_change":"2024-03-16T09:39:56.176621+0000","last_active":"2024-03-16T09:39:57.268834+0000","last_peered":"2024-03-16T09:39:57.268834+0000","last_clean":"2024-03-16T09:39:57.268834+0000","last_became_active":"2024-03-16T09:39:56.176296+0000","last_became_peered":"2024-03-16T09:39:56.176296+0000","last_unstale":"2024-03-16T09:39:57.268834+0000","last_undegraded":"2024-03-16T09:39:57.268834+0000","last_fullsized":"2024-03-16T09:39:57.268834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:54.673937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185552+0000","last_change":"2024-03-16T09:39:56.197689+0000","last_active":"2024-03-16T09:39:57.185552+0000","last_peered":"2024-03-16T09:39:57.185552+0000","last_clean":"2024-03-16T09:39:57.185552+0000","last_became_active":"2024-03-16T09:39:56.197343+0000","last_became_peered":"2024-03-16T09:39:56.197343+0000","last_unstale":"2024-03-16T09:39:57.185552+0000","last_undegraded":"2024-03-16T09:39:57.185552+0000","last_fullsized":"2024-03-16T09:39:57.185552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:28:10.218048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T09:39:56.194276+0000","last_change":"2024-03-16T09:39:56.193551+0000","last_active":"2024-03-16T09:39:56.194276+0000","last_peered":"2024-03-16T09:39:56.194276+0000","last_clean":"2024-03-16T09:39:56.194276+0000","last_became_active":"2024-03-16T09:39:56.192901+0000","last_became_peered":"2024-03-16T09:39:56.192901+0000","last_unstale":"2024-03-16T09:39:56.194276+0000","last_undegraded":"2024-03-16T09:39:56.194276+0000","last_fullsized":"2024-03-16T09:39:56.194276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:01:27.116485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.185283+0000","last_change":"2024-03-16T09:39:56.188484+0000","last_active":"2024-03-16T09:39:57.185283+0000","last_peered":"2024-03-16T09:39:57.185283+0000","last_clean":"2024-03-16T09:39:57.185283+0000","last_became_active":"2024-03-16T09:39:56.188144+0000","last_became_peered":"2024-03-16T09:39:56.188144+0000","last_unstale":"2024-03-16T09:39:57.185283+0000","last_undegraded":"2024-03-16T09:39:57.185283+0000","last_fullsized":"2024-03-16T09:39:57.185283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:01:10.042765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.181936+0000","last_change":"2024-03-16T09:39:56.187448+0000","last_active":"2024-03-16T09:39:57.181936+0000","last_peered":"2024-03-16T09:39:57.181936+0000","last_clean":"2024-03-16T09:39:57.181936+0000","last_became_active":"2024-03-16T09:39:56.186772+0000","last_became_peered":"2024-03-16T09:39:56.186772+0000","last_unstale":"2024-03-16T09:39:57.181936+0000","last_undegraded":"2024-03-16T09:39:57.181936+0000","last_fullsized":"2024-03-16T09:39:57.181936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:45:43.315011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268500+0000","last_change":"2024-03-16T09:39:56.191004+0000","last_active":"2024-03-16T09:39:57.268500+0000","last_peered":"2024-03-16T09:39:57.268500+0000","last_clean":"2024-03-16T09:39:57.268500+0000","last_became_active":"2024-03-16T09:39:56.190646+0000","last_became_peered":"2024-03-16T09:39:56.190646+0000","last_unstale":"2024-03-16T09:39:57.268500+0000","last_undegraded":"2024-03-16T09:39:57.268500+0000","last_fullsized":"2024-03-16T09:39:57.268500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:03.191129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-16T09:39:57.268901+0000","last_change":"2024-03-16T09:39:56.197552+0000","last_active":"2024-03-16T09:39:57.268901+0000","last_peered":"2024-03-16T09:39:57.268901+0000","last_clean":"2024-03-16T09:39:57.268901+0000","last_became_active":"2024-03-16T09:39:56.197215+0000","last_became_peered":"2024-03-16T09:39:56.197215+0000","last_unstale":"2024-03-16T09:39:57.268901+0000","last_undegraded":"2024-03-16T09:39:57.268901+0000","last_fullsized":"2024-03-16T09:39:57.268901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:45.795328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6424,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":124,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":16106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":913408,"data_stored":434445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1036573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":443833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":443753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1540096,"data_stored":1035292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462949,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":445283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":933888,"data_stored":443119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":443037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724587,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36832,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708320,"statfs":{"total":95995035648,"available":95957319680,"internally_reserved":0,"allocated":1519616,"data_stored":1024995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T09:40:02.783 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T09:40:02.783 INFO:tasks.cephadm:Setting up client nodes... 2024-03-16T09:40:02.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T09:40:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:02 smithi077 ceph-mon[32171]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:02 smithi077 ceph-mon[32171]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:02 smithi077 ceph-mon[32171]: pgmap v288: 97 pgs: 97 active+clean; 583 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-03-16T09:40:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:02 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/948737091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:40:03.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:02 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/999281129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:03.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:02 smithi129 ceph-mon[35841]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:03.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:02 smithi129 ceph-mon[35841]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:03.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:02 smithi129 ceph-mon[35841]: pgmap v288: 97 pgs: 97 active+clean; 583 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-03-16T09:40:03.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:02 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/948737091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:40:03.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:02 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/999281129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:03.056 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:02 smithi119 ceph-mon[35632]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:02 smithi119 ceph-mon[35632]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:40:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:02 smithi119 ceph-mon[35632]: pgmap v288: 97 pgs: 97 active+clean; 583 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-03-16T09:40:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:02 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/948737091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:40:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:02 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/999281129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:04.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:03 smithi077 ceph-mon[32171]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:40:04.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:03 smithi077 ceph-mon[32171]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:40:04.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:03 smithi129 ceph-mon[35841]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:40:04.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:03 smithi129 ceph-mon[35841]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:40:04.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:03 smithi119 ceph-mon[35632]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:40:04.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:03 smithi119 ceph-mon[35632]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:40:04.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/2894249769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 msgr2=0x7f2abc10a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/2894249769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc10a770 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400b0d0 tx=0x7f2ab400b3e0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/2894249769 shutdown_connections 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/2894249769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc10a770 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/2894249769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2abc102320 0x7f2abc102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/2894249769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2abc103520 0x7f2abc103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/2894249769 >> 172.21.15.77:0/2894249769 conn(0x7f2abc0fd8e0 msgr2=0x7f2abc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/2894249769 shutdown_connections 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/2894249769 wait complete. 2024-03-16T09:40:04.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 Processor -- start 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.651+0000 7f2ac2fb7700 1 -- start start 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac2fb7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2abc102320 0x7f2abc076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac2fb7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2abc103520 0x7f2abc0746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac2fb7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac2fb7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10dda0 con 0x7f2abc103ec0 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac2fb7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10dc20 con 0x7f2abc103520 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac2fb7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10df20 con 0x7f2abc102320 2024-03-16T09:40:04.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac1554700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac0d53700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2abc102320 0x7f2abc076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac1554700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59818/0 (socket says 172.21.15.77:59818) 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac1554700 1 -- 172.21.15.77:0/1946665900 learned_addr learned my addr 172.21.15.77:0/1946665900 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2abbfff700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2abc103520 0x7f2abc0746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac0d53700 1 -- 172.21.15.77:0/1946665900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2abc103520 msgr2=0x7f2abc0746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac0d53700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2abc103520 0x7f2abc0746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac0d53700 1 -- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 msgr2=0x7f2abc074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac0d53700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac0d53700 1 -- 172.21.15.77:0/1946665900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc0754b0 con 0x7f2abc102320 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac1554700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2abbfff700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2abc103520 0x7f2abc0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:04.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac0d53700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2abc102320 0x7f2abc076070 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000e9c0 tx=0x7f2ab000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ab9ffb700 1 -- 172.21.15.77:0/1946665900 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0004220 con 0x7f2abc102320 2024-03-16T09:40:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ab9ffb700 1 -- 172.21.15.77:0/1946665900 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ab00043c0 con 0x7f2abc102320 2024-03-16T09:40:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc1a2460 con 0x7f2abc102320 2024-03-16T09:40:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.652+0000 7f2ab9ffb700 1 -- 172.21.15.77:0/1946665900 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0003c40 con 0x7f2abc102320 2024-03-16T09:40:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.653+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc1a2970 con 0x7f2abc102320 2024-03-16T09:40:04.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.653+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc0507e0 con 0x7f2abc102320 2024-03-16T09:40:04.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.655+0000 7f2ab9ffb700 1 -- 172.21.15.77:0/1946665900 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2ab000c7c0 con 0x7f2abc102320 2024-03-16T09:40:04.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.655+0000 7f2ab9ffb700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aac0a29d0 0x7f2aac0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:04.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.655+0000 7f2abbfff700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aac0a29d0 0x7f2aac0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:04.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.655+0000 7f2ab9ffb700 1 -- 172.21.15.77:0/1946665900 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2ab00cec90 con 0x7f2abc102320 2024-03-16T09:40:04.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.655+0000 7f2abbfff700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aac0a29d0 0x7f2aac0a4e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8011490 tx=0x7f2aa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:04.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.659+0000 7f2ab9ffb700 1 -- 172.21.15.77:0/1946665900 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2ab009bd80 con 0x7f2abc102320 2024-03-16T09:40:04.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.932+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f2abc02db70 con 0x7f2abc102320 2024-03-16T09:40:04.935 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:04 smithi077 ceph-mon[32171]: pgmap v289: 97 pgs: 97 active+clean; 584 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-16T09:40:04.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.936+0000 7f2ab9ffb700 1 -- 172.21.15.77:0/1946665900 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) ==== 170+0+59 (secure 0 0 0) 0x7f2ab009b680 con 0x7f2abc102320 2024-03-16T09:40:04.939 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-16T09:40:04.939 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD0aPVlrlKtNxAAlKaM9t4wjYpw2jD2LOCzeQ== 2024-03-16T09:40:04.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aac0a29d0 msgr2=0x7f2aac0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:04.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aac0a29d0 0x7f2aac0a4e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8011490 tx=0x7f2aa8009260 comp rx=0 tx=0).stop 2024-03-16T09:40:04.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2abc102320 msgr2=0x7f2abc076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:04.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2abc102320 0x7f2abc076070 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000e9c0 tx=0x7f2ab000ecd0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 shutdown_connections 2024-03-16T09:40:04.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aac0a29d0 0x7f2aac0a4e90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2abc103ec0 0x7f2abc074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2abc102320 0x7f2abc076070 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 --2- 172.21.15.77:0/1946665900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2abc103520 0x7f2abc0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:04.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 >> 172.21.15.77:0/1946665900 conn(0x7f2abc0fd8e0 msgr2=0x7f2abc1067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:04.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 shutdown_connections 2024-03-16T09:40:04.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:04.939+0000 7f2ac2fb7700 1 -- 172.21.15.77:0/1946665900 wait complete. 2024-03-16T09:40:05.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:04 smithi129 ceph-mon[35841]: pgmap v289: 97 pgs: 97 active+clean; 584 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-16T09:40:05.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:04 smithi119 ceph-mon[35632]: pgmap v289: 97 pgs: 97 active+clean; 584 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-16T09:40:05.599 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:40:05.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-16T09:40:05.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T09:40:05.642 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T09:40:05.642 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-16T09:40:05.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-16T09:40:05.922 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:05 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1946665900' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T09:40:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:05 smithi077 ceph-mon[32171]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T09:40:06.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:05 smithi077 ceph-mon[32171]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T09:40:06.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:05 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1946665900' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T09:40:06.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:05 smithi129 ceph-mon[35841]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T09:40:06.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:05 smithi129 ceph-mon[35841]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T09:40:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:05 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1946665900' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T09:40:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:05 smithi119 ceph-mon[35632]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T09:40:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:05 smithi119 ceph-mon[35632]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T09:40:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:06 smithi077 ceph-mon[32171]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-16T09:40:07.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:06 smithi129 ceph-mon[35841]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-16T09:40:07.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:06 smithi119 ceph-mon[35632]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-16T09:40:07.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 -- 172.21.15.77:0/2861651102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78108300 msgr2=0x7f1c781086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:07.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 --2- 172.21.15.77:0/2861651102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78108300 0x7f1c781086e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1c680099c0 tx=0x7f1c68009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 -- 172.21.15.77:0/2861651102 shutdown_connections 2024-03-16T09:40:07.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 --2- 172.21.15.77:0/2861651102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c78102ce0 0x7f1c7810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 --2- 172.21.15.77:0/2861651102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78108300 0x7f1c781086e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 --2- 172.21.15.77:0/2861651102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c78102320 0x7f1c781027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 -- 172.21.15.77:0/2861651102 >> 172.21.15.77:0/2861651102 conn(0x7f1c78079400 msgr2=0x7f1c78079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 -- 172.21.15.77:0/2861651102 shutdown_connections 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.631+0000 7f1c80237700 1 -- 172.21.15.77:0/2861651102 wait complete. 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 Processor -- start 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 -- start start 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78102320 0x7f1c7819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c78102ce0 0x7f1c7819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c78108300 0x7f1c78198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c78112430 con 0x7f1c78102320 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1c781122b0 con 0x7f1c78102ce0 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c80237700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1c781125b0 con 0x7f1c78108300 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c7dfd3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78102320 0x7f1c7819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:07.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c7dfd3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78102320 0x7f1c7819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60032/0 (socket says 172.21.15.77:60032) 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c7dfd3700 1 -- 172.21.15.77:0/3229899111 learned_addr learned my addr 172.21.15.77:0/3229899111 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c7d7d2700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c78102ce0 0x7f1c7819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.632+0000 7f1c7e7d4700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c78108300 0x7f1c78198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c7dfd3700 1 -- 172.21.15.77:0/3229899111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c78108300 msgr2=0x7f1c78198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c7dfd3700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c78108300 0x7f1c78198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c7dfd3700 1 -- 172.21.15.77:0/3229899111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c78102ce0 msgr2=0x7f1c7819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c7dfd3700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c78102ce0 0x7f1c7819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c7dfd3700 1 -- 172.21.15.77:0/3229899111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c78199010 con 0x7f1c78102320 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c7dfd3700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78102320 0x7f1c7819e610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1c68006010 tx=0x7f1c680055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c6effd700 1 -- 172.21.15.77:0/3229899111 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68026070 con 0x7f1c78102320 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c781992a0 con 0x7f1c78102320 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c6effd700 1 -- 172.21.15.77:0/3229899111 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c6800f460 con 0x7f1c78102320 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c6effd700 1 -- 172.21.15.77:0/3229899111 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68009e60 con 0x7f1c78102320 2024-03-16T09:40:07.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.633+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c781a53c0 con 0x7f1c78102320 2024-03-16T09:40:07.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.634+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c781039b0 con 0x7f1c78102320 2024-03-16T09:40:07.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.635+0000 7f1c6effd700 1 -- 172.21.15.77:0/3229899111 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1c68029070 con 0x7f1c78102320 2024-03-16T09:40:07.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.636+0000 7f1c6effd700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c640a2920 0x7f1c640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:07.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.636+0000 7f1c7d7d2700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c640a2920 0x7f1c640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:07.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.636+0000 7f1c6effd700 1 -- 172.21.15.77:0/3229899111 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1c680cf640 con 0x7f1c78102320 2024-03-16T09:40:07.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.636+0000 7f1c7d7d2700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c640a2920 0x7f1c640a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c78199ea0 tx=0x7f1c740073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:07.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.639+0000 7f1c6effd700 1 -- 172.21.15.77:0/3229899111 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c68030020 con 0x7f1c78102320 2024-03-16T09:40:07.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.895+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f1c7802db70 con 0x7f1c78102320 2024-03-16T09:40:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.900+0000 7f1c6effd700 1 -- 172.21.15.77:0/3229899111 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262660 (secure 0 0 0) 0x7f1c68013070 con 0x7f1c78102320 2024-03-16T09:40:07.907 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c640a2920 msgr2=0x7f1c640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c640a2920 0x7f1c640a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c78199ea0 tx=0x7f1c740073d0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78102320 msgr2=0x7f1c7819e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78102320 0x7f1c7819e610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1c68006010 tx=0x7f1c680055f0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 shutdown_connections 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c640a2920 0x7f1c640a4de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c78108300 0x7f1c78198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c78102320 0x7f1c7819e610 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 --2- 172.21.15.77:0/3229899111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c78102ce0 0x7f1c7819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:07.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 >> 172.21.15.77:0/3229899111 conn(0x7f1c78079400 msgr2=0x7f1c78101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:07.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 shutdown_connections 2024-03-16T09:40:07.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:07.903+0000 7f1c80237700 1 -- 172.21.15.77:0/3229899111 wait complete. 2024-03-16T09:40:08.598 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3114858521},{"type":"v1","addr":"172.21.15.77:6801","nonce":3114858521}]},"active_addr":"172.21.15.77:6801/3114858521","active_change":"2024-03-16T09:32:57.693181+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.77:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1858843655}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2486423961}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3932809728}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":510540831}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2731270882}]}]} 2024-03-16T09:40:08.604 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-16T09:40:08.604 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-16T09:40:08.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T09:40:08.846 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:09.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:08 smithi077 ceph-mon[32171]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-16T09:40:09.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:08 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3229899111' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T09:40:09.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:08 smithi119 ceph-mon[35632]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-16T09:40:09.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:08 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3229899111' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T09:40:09.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:08 smithi129 ceph-mon[35841]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-16T09:40:09.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:08 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3229899111' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T09:40:10.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.409+0000 7f6b8532e700 1 -- 172.21.15.77:0/1264833874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b80101b70 msgr2=0x7f6b8010e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.409+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1264833874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b80101b70 0x7f6b8010e010 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800b0a0 tx=0x7f6b7800b3b0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.409+0000 7f6b8532e700 1 -- 172.21.15.77:0/1264833874 shutdown_connections 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.409+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1264833874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b80101b70 0x7f6b8010e010 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.409+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1264833874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b80107260 0x7f6b80107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.410+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1264833874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801011b0 0x7f6b80101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.410+0000 7f6b8532e700 1 -- 172.21.15.77:0/1264833874 >> 172.21.15.77:0/1264833874 conn(0x7f6b800781e0 msgr2=0x7f6b800785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.410+0000 7f6b8532e700 1 -- 172.21.15.77:0/1264833874 shutdown_connections 2024-03-16T09:40:10.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.410+0000 7f6b8532e700 1 -- 172.21.15.77:0/1264833874 wait complete. 2024-03-16T09:40:10.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.410+0000 7f6b8532e700 1 Processor -- start 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.410+0000 7f6b8532e700 1 -- start start 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b801011b0 0x7f6b8019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80101b70 0x7f6b8019d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b80107260 0x7f6b801975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80111340 con 0x7f6b801011b0 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6b801111c0 con 0x7f6b80101b70 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b801114c0 con 0x7f6b80107260 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b801011b0 0x7f6b8019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b801011b0 0x7f6b8019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60042/0 (socket says 172.21.15.77:60042) 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 -- 172.21.15.77:0/1655231566 learned_addr learned my addr 172.21.15.77:0/1655231566 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7f7fe700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b80107260 0x7f6b801975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7e7fc700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80101b70 0x7f6b8019d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 -- 172.21.15.77:0/1655231566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b80107260 msgr2=0x7f6b801975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b80107260 0x7f6b801975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 -- 172.21.15.77:0/1655231566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80101b70 msgr2=0x7f6b8019d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80101b70 0x7f6b8019d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 -- 172.21.15.77:0/1655231566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b80197d40 con 0x7f6b801011b0 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b7effd700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b801011b0 0x7f6b8019d420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000e9c0 tx=0x7f6b7000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b67fff700 1 -- 172.21.15.77:0/1655231566 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7000cbd0 con 0x7f6b801011b0 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b67fff700 1 -- 172.21.15.77:0/1655231566 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b7000ee60 con 0x7f6b801011b0 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b80198030 con 0x7f6b801011b0 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b67fff700 1 -- 172.21.15.77:0/1655231566 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b700055b0 con 0x7f6b801011b0 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.411+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8006a9f0 con 0x7f6b801011b0 2024-03-16T09:40:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.412+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b80068f90 con 0x7f6b801011b0 2024-03-16T09:40:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.414+0000 7f6b67fff700 1 -- 172.21.15.77:0/1655231566 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6b70004750 con 0x7f6b801011b0 2024-03-16T09:40:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.414+0000 7f6b67fff700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b680a29a0 0x7f6b680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.414+0000 7f6b7e7fc700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b680a29a0 0x7f6b680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.414+0000 7f6b67fff700 1 -- 172.21.15.77:0/1655231566 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f6b700cee40 con 0x7f6b801011b0 2024-03-16T09:40:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.414+0000 7f6b7e7fc700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b680a29a0 0x7f6b680a4e60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b80078920 tx=0x7f6b74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:10.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.417+0000 7f6b67fff700 1 -- 172.21.15.77:0/1655231566 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b70053740 con 0x7f6b801011b0 2024-03-16T09:40:10.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.612+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b80198b10 con 0x7f6b801011b0 2024-03-16T09:40:10.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.613+0000 7f6b67fff700 1 -- 172.21.15.77:0/1655231566 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22231 (secure 0 0 0) 0x7f6b7009bf30 con 0x7f6b801011b0 2024-03-16T09:40:10.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:40:10.617 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":86,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:57.172185+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b680a29a0 msgr2=0x7f6b680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b680a29a0 0x7f6b680a4e60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b80078920 tx=0x7f6b74009260 comp rx=0 tx=0).stop 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b801011b0 msgr2=0x7f6b8019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b801011b0 0x7f6b8019d420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000e9c0 tx=0x7f6b7000ecd0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 shutdown_connections 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6b680a29a0 0x7f6b680a4e60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b80107260 0x7f6b801975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b801011b0 0x7f6b8019d420 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 --2- 172.21.15.77:0/1655231566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80101b70 0x7f6b8019d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 >> 172.21.15.77:0/1655231566 conn(0x7f6b800781e0 msgr2=0x7f6b801000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 shutdown_connections 2024-03-16T09:40:10.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:10.616+0000 7f6b8532e700 1 -- 172.21.15.77:0/1655231566 wait complete. 2024-03-16T09:40:10.992 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:10 smithi077 ceph-mon[32171]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-03-16T09:40:10.992 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:10 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1655231566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:11.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:10 smithi119 ceph-mon[35632]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-03-16T09:40:11.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:10 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1655231566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:11.287 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-16T09:40:11.287 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T09:40:11.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:10 smithi129 ceph-mon[35841]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-03-16T09:40:11.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:10 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1655231566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:11.528 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:13.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 -- 172.21.15.77:0/925977801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac102ce0 msgr2=0x7f50ac10f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/925977801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac102ce0 0x7f50ac10f180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f509c009990 tx=0x7f509c009ca0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 -- 172.21.15.77:0/925977801 shutdown_connections 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/925977801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac1083d0 0x7f50ac1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/925977801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac102ce0 0x7f50ac10f180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/925977801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102320 0x7f50ac1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 -- 172.21.15.77:0/925977801 >> 172.21.15.77:0/925977801 conn(0x7f50ac079400 msgr2=0x7f50ac079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 -- 172.21.15.77:0/925977801 shutdown_connections 2024-03-16T09:40:13.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.082+0000 7f50b1bf0700 1 -- 172.21.15.77:0/925977801 wait complete. 2024-03-16T09:40:13.087 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:12 smithi077 ceph-mon[32171]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-16T09:40:13.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 Processor -- start 2024-03-16T09:40:13.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 -- start start 2024-03-16T09:40:13.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac102320 0x7f50ac19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:13.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102ce0 0x7f50ac19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac1083d0 0x7f50ac198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50ac1124c0 con 0x7f50ac1083d0 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f50ac112340 con 0x7f50ac102320 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.083+0000 7f50b1bf0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f50ac112640 con 0x7f50ac102ce0 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50ab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac102320 0x7f50ac19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac1083d0 0x7f50ac198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102ce0 0x7f50ac19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50ab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac102320 0x7f50ac19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:47860/0 (socket says 172.21.15.77:47860) 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac1083d0 0x7f50ac198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60058/0 (socket says 172.21.15.77:60058) 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102ce0 0x7f50ac19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:41596/0 (socket says 172.21.15.77:41596) 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50ab7fe700 1 -- 172.21.15.77:0/769647017 learned_addr learned my addr 172.21.15.77:0/769647017 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:13.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 -- 172.21.15.77:0/769647017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac102320 msgr2=0x7f50ac19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac102320 0x7f50ac19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 -- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac1083d0 msgr2=0x7f50ac198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac1083d0 0x7f50ac198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 -- 172.21.15.77:0/769647017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ac199020 con 0x7f50ac102ce0 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50ab7fe700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac102320 0x7f50ac19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50abfff700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac1083d0 0x7f50ac198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50aaffd700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102ce0 0x7f50ac19ebb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f509400e9f0 tx=0x7f509400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50a8ff9700 1 -- 172.21.15.77:0/769647017 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5094004220 con 0x7f50ac102ce0 2024-03-16T09:40:13.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ac199310 con 0x7f50ac102ce0 2024-03-16T09:40:13.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50a8ff9700 1 -- 172.21.15.77:0/769647017 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50940043c0 con 0x7f50ac102ce0 2024-03-16T09:40:13.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50a8ff9700 1 -- 172.21.15.77:0/769647017 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5094005020 con 0x7f50ac102ce0 2024-03-16T09:40:13.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.084+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ac1a55f0 con 0x7f50ac102ce0 2024-03-16T09:40:13.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.085+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ac103a10 con 0x7f50ac102ce0 2024-03-16T09:40:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.086+0000 7f50a8ff9700 1 -- 172.21.15.77:0/769647017 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5094004750 con 0x7f50ac102ce0 2024-03-16T09:40:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.087+0000 7f50a8ff9700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50980a2920 0x7f50980a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.087+0000 7f50ab7fe700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50980a2920 0x7f50980a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.087+0000 7f50a8ff9700 1 -- 172.21.15.77:0/769647017 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f50940cea60 con 0x7f50ac102ce0 2024-03-16T09:40:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.087+0000 7f50ab7fe700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50980a2920 0x7f50980a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f50a4009fd0 tx=0x7f50a4009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:13.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.092+0000 7f50a8ff9700 1 -- 172.21.15.77:0/769647017 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f509409ba90 con 0x7f50ac102ce0 2024-03-16T09:40:13.102 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:12 smithi129 ceph-mon[35841]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-16T09:40:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:12 smithi119 ceph-mon[35632]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-16T09:40:13.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.290+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50ac199c70 con 0x7f50ac102ce0 2024-03-16T09:40:13.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.293+0000 7f50a8ff9700 1 -- 172.21.15.77:0/769647017 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22231 (secure 0 0 0) 0x7f509409b390 con 0x7f50ac102ce0 2024-03-16T09:40:13.296 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:40:13.296 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":86,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:39:57.172185+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50980a2920 msgr2=0x7f50980a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50980a2920 0x7f50980a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f50a4009fd0 tx=0x7f50a4009040 comp rx=0 tx=0).stop 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102ce0 msgr2=0x7f50ac19ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102ce0 0x7f50ac19ebb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f509400e9f0 tx=0x7f509400ed00 comp rx=0 tx=0).stop 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 shutdown_connections 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50980a2920 0x7f50980a4de0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50ac102320 0x7f50ac19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50ac1083d0 0x7f50ac198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 --2- 172.21.15.77:0/769647017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50ac102ce0 0x7f50ac19ebb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:13.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 >> 172.21.15.77:0/769647017 conn(0x7f50ac079400 msgr2=0x7f50ac101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 shutdown_connections 2024-03-16T09:40:13.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:13.295+0000 7f50b1bf0700 1 -- 172.21.15.77:0/769647017 wait complete. 2024-03-16T09:40:13.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-16T09:40:13.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-16T09:40:13.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-16T09:40:13.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-16T09:40:13.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-16T09:40:13.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-16T09:40:13.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-16T09:40:13.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-16T09:40:13.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-16T09:40:13.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-16T09:40:13.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-16T09:40:13.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-16T09:40:14.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:13 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/769647017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:14.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:13 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/769647017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:14.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:13 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/769647017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:40:14.607 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:14.650 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:14.715 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:14.716 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:15.037 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:14 smithi077 ceph-mon[32171]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-03-16T09:40:15.042 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:15.043 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:14 smithi119 ceph-mon[35632]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-03-16T09:40:15.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:14 smithi129 ceph-mon[35841]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.4 KiB/s wr, 8 op/s 2024-03-16T09:40:15.674 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:15.674 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:15.674 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:15.675 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:17.074 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:16 smithi077 ceph-mon[32171]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-03-16T09:40:17.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:16 smithi119 ceph-mon[35632]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-03-16T09:40:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:16 smithi129 ceph-mon[35841]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-03-16T09:40:18.950 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:18 smithi077 ceph-mon[32171]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-16T09:40:19.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:18 smithi119 ceph-mon[35632]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-16T09:40:19.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:18 smithi129 ceph-mon[35841]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-16T09:40:21.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:20 smithi077 ceph-mon[32171]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-16T09:40:21.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:20 smithi119 ceph-mon[35632]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-16T09:40:21.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:20 smithi129 ceph-mon[35841]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-16T09:40:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.119+0000 7f258feb9700 1 -- 172.21.15.77:0/1329596119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588108320 msgr2=0x7f2588108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.119+0000 7f258feb9700 1 --2- 172.21.15.77:0/1329596119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588108320 0x7f2588108700 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f25840099c0 tx=0x7f2584009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 -- 172.21.15.77:0/1329596119 shutdown_connections 2024-03-16T09:40:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 --2- 172.21.15.77:0/1329596119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f258806a920 0x7f258810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 --2- 172.21.15.77:0/1329596119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588108320 0x7f2588108700 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 --2- 172.21.15.77:0/1329596119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2588069f80 0x7f258806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 -- 172.21.15.77:0/1329596119 >> 172.21.15.77:0/1329596119 conn(0x7f2588078100 msgr2=0x7f2588078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:22.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 -- 172.21.15.77:0/1329596119 shutdown_connections 2024-03-16T09:40:22.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 -- 172.21.15.77:0/1329596119 wait complete. 2024-03-16T09:40:22.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 Processor -- start 2024-03-16T09:40:22.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.120+0000 7f258feb9700 1 -- start start 2024-03-16T09:40:22.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588069f80 0x7f258819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f258806a920 0x7f258819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2588108320 0x7f25881987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2588112480 con 0x7f2588069f80 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2588112300 con 0x7f2588108320 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2588112600 con 0x7f258806a920 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588069f80 0x7f258819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588069f80 0x7f258819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60620/0 (socket says 172.21.15.77:60620) 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 -- 172.21.15.77:0/2428064034 learned_addr learned my addr 172.21.15.77:0/2428064034 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258d454700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f258806a920 0x7f258819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258e456700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2588108320 0x7f25881987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 -- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f258806a920 msgr2=0x7f258819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f258806a920 0x7f258819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 -- 172.21.15.77:0/2428064034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2588108320 msgr2=0x7f25881987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2588108320 0x7f25881987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 -- 172.21.15.77:0/2428064034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2588198fa0 con 0x7f2588069f80 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258d454700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f258806a920 0x7f258819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258dc55700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588069f80 0x7f258819e640 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2584005910 tx=0x7f2584004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584026070 con 0x7f2588069f80 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2588199230 con 0x7f2588069f80 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2584003c70 con 0x7f2588069f80 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258400fa40 con 0x7f2588069f80 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.121+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25881a54a0 con 0x7f2588069f80 2024-03-16T09:40:22.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.122+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2588062a60 con 0x7f2588069f80 2024-03-16T09:40:22.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.123+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2584029070 con 0x7f2588069f80 2024-03-16T09:40:22.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f257effd700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25740a2920 0x7f25740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f258d454700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25740a2920 0x7f25740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f25840cf9e0 con 0x7f2588069f80 2024-03-16T09:40:22.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f257effd700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] conn(0x7f25740ac100 0x7f25740ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 --> [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25740aebd0 con 0x7f25740ac100 2024-03-16T09:40:22.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f25840cfec0 con 0x7f2588069f80 2024-03-16T09:40:22.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f258d454700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25740a2920 0x7f25740a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25880787e0 tx=0x7f2578009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:22.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.124+0000 7f258e456700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] conn(0x7f25740ac100 0x7f25740ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.125+0000 7f258e456700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] conn(0x7f25740ac100 0x7f25740ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:22.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.125+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== osd.8 v2:172.21.15.129:6816/3565481849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f25740aebd0 con 0x7f25740ac100 2024-03-16T09:40:22.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 --> [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2588062e60 con 0x7f25740ac100 2024-03-16T09:40:22.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f257effd700 1 -- 172.21.15.77:0/2428064034 <== osd.8 v2:172.21.15.129:6816/3565481849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2588062e60 con 0x7f25740ac100 2024-03-16T09:40:22.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] conn(0x7f25740ac100 msgr2=0x7f25740ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] conn(0x7f25740ac100 0x7f25740ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25740a2920 msgr2=0x7f25740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25740a2920 0x7f25740a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25880787e0 tx=0x7f2578009260 comp rx=0 tx=0).stop 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588069f80 msgr2=0x7f258819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 --2- 172.21.15.77:0/2428064034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2588069f80 0x7f258819e640 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2584005910 tx=0x7f2584004270 comp rx=0 tx=0).stop 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258e456700 1 -- 172.21.15.77:0/2428064034 reap_dead start 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 shutdown_connections 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 >> 172.21.15.77:0/2428064034 conn(0x7f2588078100 msgr2=0x7f25880fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.142+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 shutdown_connections 2024-03-16T09:40:22.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.143+0000 7f258feb9700 1 -- 172.21.15.77:0/2428064034 wait complete. 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 -- 172.21.15.77:0/3149533694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 msgr2=0x7f7700101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 --2- 172.21.15.77:0/3149533694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 0x7f7700101630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009990 tx=0x7f76f4009ca0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 -- 172.21.15.77:0/3149533694 shutdown_connections 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 --2- 172.21.15.77:0/3149533694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7700101b70 0x7f770010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 --2- 172.21.15.77:0/3149533694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7700107210 0x7f77001075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 --2- 172.21.15.77:0/3149533694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 0x7f7700101630 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 -- 172.21.15.77:0/3149533694 >> 172.21.15.77:0/3149533694 conn(0x7f7700078100 msgr2=0x7f7700078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 -- 172.21.15.77:0/3149533694 shutdown_connections 2024-03-16T09:40:22.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 -- 172.21.15.77:0/3149533694 wait complete. 2024-03-16T09:40:22.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.734+0000 7f7705647700 1 Processor -- start 2024-03-16T09:40:22.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f7705647700 1 -- start start 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f7705647700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 0x7f77001997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f7705647700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7700101b70 0x7f7700199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f7705647700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7700107210 0x7f77001a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f7705647700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7700111320 con 0x7f77001011d0 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f7705647700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f77001111a0 con 0x7f7700101b70 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f7705647700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77001114a0 con 0x7f7700107210 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 0x7f77001997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 0x7f77001997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60644/0 (socket says 172.21.15.77:60644) 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 -- 172.21.15.77:0/2330774079 learned_addr learned my addr 172.21.15.77:0/2330774079 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76ff7fe700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7700107210 0x7f77001a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76fe7fc700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7700101b70 0x7f7700199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 -- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7700107210 msgr2=0x7f77001a4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7700107210 0x7f77001a4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 -- 172.21.15.77:0/2330774079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7700101b70 msgr2=0x7f7700199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7700101b70 0x7f7700199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 -- 172.21.15.77:0/2330774079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77001a4d60 con 0x7f77001011d0 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76ff7fe700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7700107210 0x7f77001a4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76fe7fc700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7700101b70 0x7f7700199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.735+0000 7f76feffd700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 0x7f77001997a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f76f000e9a0 tx=0x7f76f000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.736+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f000ca10 con 0x7f77001011d0 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.736+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76f000cbb0 con 0x7f77001011d0 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.736+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77001a4ff0 con 0x7f77001011d0 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.736+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0018680 con 0x7f77001011d0 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.736+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77001a5440 con 0x7f77001011d0 2024-03-16T09:40:22.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.736+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77000629e0 con 0x7f77001011d0 2024-03-16T09:40:22.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.738+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f76f0010430 con 0x7f77001011d0 2024-03-16T09:40:22.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.738+0000 7f76e7fff700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f76e80a29a0 0x7f76e80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76fe7fc700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f76e80a29a0 0x7f76e80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f76f00cf590 con 0x7f77001011d0 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76e7fff700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] conn(0x7f76e80ac180 0x7f76e80ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 --> [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76f0020e50 con 0x7f76e80ac180 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f76f00cfa70 con 0x7f77001011d0 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76fe7fc700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f76e80a29a0 0x7f76e80a4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f76f40098e0 tx=0x7f76f4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76ff7fe700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] conn(0x7f76e80ac180 0x7f76e80ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.739+0000 7f76ff7fe700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] conn(0x7f76e80ac180 0x7f76e80ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.740+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== osd.2 v2:172.21.15.129:6800/2888466595 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f76f0020e50 con 0x7f76e80ac180 2024-03-16T09:40:22.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.756+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 --> [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7700062e10 con 0x7f76e80ac180 2024-03-16T09:40:22.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.756+0000 7f76e7fff700 1 -- 172.21.15.77:0/2330774079 <== osd.2 v2:172.21.15.129:6800/2888466595 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7700062e10 con 0x7f76e80ac180 2024-03-16T09:40:22.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] conn(0x7f76e80ac180 msgr2=0x7f76e80ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] conn(0x7f76e80ac180 0x7f76e80ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f76e80a29a0 msgr2=0x7f76e80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f76e80a29a0 0x7f76e80a4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f76f40098e0 tx=0x7f76f4005960 comp rx=0 tx=0).stop 2024-03-16T09:40:22.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 msgr2=0x7f77001997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:22.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 --2- 172.21.15.77:0/2330774079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77001011d0 0x7f77001997a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f76f000e9a0 tx=0x7f76f000ecb0 comp rx=0 tx=0).stop 2024-03-16T09:40:22.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f76ff7fe700 1 -- 172.21.15.77:0/2330774079 reap_dead start 2024-03-16T09:40:22.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 shutdown_connections 2024-03-16T09:40:22.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 >> 172.21.15.77:0/2330774079 conn(0x7f7700078100 msgr2=0x7f77000fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:22.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 shutdown_connections 2024-03-16T09:40:22.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:22.757+0000 7f7705647700 1 -- 172.21.15.77:0/2330774079 wait complete. 2024-03-16T09:40:23.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:22 smithi077 ceph-mon[32171]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T09:40:23.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:22 smithi119 ceph-mon[35632]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T09:40:23.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:22 smithi129 ceph-mon[35841]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T09:40:24.817 INFO:teuthology.orchestra.run.smithi077.stdout:77309411391 2024-03-16T09:40:24.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-16T09:40:24.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 -- 172.21.15.77:0/365632769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c1025e0 msgr2=0x7f259c105e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:24.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 --2- 172.21.15.77:0/365632769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c1025e0 0x7f259c105e70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f25900099c0 tx=0x7f2590009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:24.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 -- 172.21.15.77:0/365632769 shutdown_connections 2024-03-16T09:40:24.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 --2- 172.21.15.77:0/365632769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f259c106550 0x7f259c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:24.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 --2- 172.21.15.77:0/365632769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c101c30 0x7f259c102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:24.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 --2- 172.21.15.77:0/365632769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c1025e0 0x7f259c105e70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:24.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 -- 172.21.15.77:0/365632769 >> 172.21.15.77:0/365632769 conn(0x7f259c0777d0 msgr2=0x7f259c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:24.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 -- 172.21.15.77:0/365632769 shutdown_connections 2024-03-16T09:40:24.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.976+0000 7f25a1b0e700 1 -- 172.21.15.77:0/365632769 wait complete. 2024-03-16T09:40:24.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.977+0000 7f25a1b0e700 1 Processor -- start 2024-03-16T09:40:24.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.977+0000 7f25a1b0e700 1 -- start start 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f25a1b0e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c101c30 0x7f259c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f25a1b0e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f259c1025e0 0x7f259c19dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f25a1b0e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c106550 0x7f259c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f25a1b0e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f259c1113e0 con 0x7f259c106550 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f25a1b0e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f259c111260 con 0x7f259c1025e0 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f25a1b0e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f259c111560 con 0x7f259c101c30 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c106550 0x7f259c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c106550 0x7f259c197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60668/0 (socket says 172.21.15.77:60668) 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 -- 172.21.15.77:0/960527669 learned_addr learned my addr 172.21.15.77:0/960527669 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259affd700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f259c1025e0 0x7f259c19dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259b7fe700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c101c30 0x7f259c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:24.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 -- 172.21.15.77:0/960527669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c101c30 msgr2=0x7f259c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c101c30 0x7f259c19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 -- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f259c1025e0 msgr2=0x7f259c19dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f259c1025e0 0x7f259c19dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 -- 172.21.15.77:0/960527669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259c197fa0 con 0x7f259c106550 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259affd700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f259c1025e0 0x7f259c19dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f259bfff700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c106550 0x7f259c197740 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f259400ef30 tx=0x7f259400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25940092e0 con 0x7f259c106550 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.978+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2594004bb0 con 0x7f259c106550 2024-03-16T09:40:24.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.979+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259c198230 con 0x7f259c106550 2024-03-16T09:40:24.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.979+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2594004020 con 0x7f259c106550 2024-03-16T09:40:24.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.979+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f259c1008c0 con 0x7f259c106550 2024-03-16T09:40:24.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.980+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f259c100d70 con 0x7f259c106550 2024-03-16T09:40:24.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.981+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f259401b030 con 0x7f259c106550 2024-03-16T09:40:24.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f2598ff9700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25840a29a0 0x7f25840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:24.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f259b7fe700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25840a29a0 0x7f25840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:24.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f25940ceec0 con 0x7f259c106550 2024-03-16T09:40:24.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f2598ff9700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] conn(0x7f25840ac180 0x7f25840ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:24.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 --> [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f259401fe50 con 0x7f25840ac180 2024-03-16T09:40:24.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f25940cf270 con 0x7f259c106550 2024-03-16T09:40:24.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f259b7fe700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25840a29a0 0x7f25840a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f258c009c60 tx=0x7f258c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:24.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.982+0000 7f259affd700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] conn(0x7f25840ac180 0x7f25840ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:24.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.983+0000 7f259affd700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] conn(0x7f25840ac180 0x7f25840ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:24.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:24.984+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== osd.1 v2:172.21.15.119:6800/2451540128 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f259401fe50 con 0x7f25840ac180 2024-03-16T09:40:25.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.001+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 --> [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f259c1011f0 con 0x7f25840ac180 2024-03-16T09:40:25.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.001+0000 7f2598ff9700 1 -- 172.21.15.77:0/960527669 <== osd.1 v2:172.21.15.119:6800/2451540128 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f259c1011f0 con 0x7f25840ac180 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.001+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] conn(0x7f25840ac180 msgr2=0x7f25840ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.001+0000 7f25a1b0e700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] conn(0x7f25840ac180 0x7f25840ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25840a29a0 msgr2=0x7f25840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f25840a29a0 0x7f25840a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f258c009c60 tx=0x7f258c009260 comp rx=0 tx=0).stop 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c106550 msgr2=0x7f259c197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 --2- 172.21.15.77:0/960527669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f259c106550 0x7f259c197740 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f259400ef30 tx=0x7f259400c3a0 comp rx=0 tx=0).stop 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f259bfff700 1 -- 172.21.15.77:0/960527669 reap_dead start 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 shutdown_connections 2024-03-16T09:40:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 >> 172.21.15.77:0/960527669 conn(0x7f259c0777d0 msgr2=0x7f259c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:25.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 shutdown_connections 2024-03-16T09:40:25.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:25.002+0000 7f25a1b0e700 1 -- 172.21.15.77:0/960527669 wait complete. 2024-03-16T09:40:25.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:24 smithi077 ceph-mon[32171]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-16T09:40:25.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:24 smithi119 ceph-mon[35632]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-16T09:40:25.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:24 smithi129 ceph-mon[35841]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-16T09:40:27.141 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:26 smithi077 ceph-mon[32171]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-16T09:40:27.168 INFO:teuthology.orchestra.run.smithi077.stdout:257698037794 2024-03-16T09:40:27.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-16T09:40:27.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:26 smithi119 ceph-mon[35632]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-16T09:40:27.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:26 smithi129 ceph-mon[35841]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-16T09:40:28.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.129+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/1359959316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc41011d0 msgr2=0x7f2bc4101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:28.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.129+0000 7f2bcbb7d700 1 --2- 172.21.15.77:0/1359959316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc41011d0 0x7f2bc4101630 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009990 tx=0x7f2bb4009ca0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.130+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/1359959316 shutdown_connections 2024-03-16T09:40:28.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.130+0000 7f2bcbb7d700 1 --2- 172.21.15.77:0/1359959316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bc4101b70 0x7f2bc410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.130+0000 7f2bcbb7d700 1 --2- 172.21.15.77:0/1359959316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bc4107210 0x7f2bc41075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.130+0000 7f2bcbb7d700 1 --2- 172.21.15.77:0/1359959316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc41011d0 0x7f2bc4101630 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.130+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/1359959316 >> 172.21.15.77:0/1359959316 conn(0x7f2bc4078100 msgr2=0x7f2bc4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:28.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.130+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/1359959316 shutdown_connections 2024-03-16T09:40:28.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.130+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/1359959316 wait complete. 2024-03-16T09:40:28.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 Processor -- start 2024-03-16T09:40:28.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 -- start start 2024-03-16T09:40:28.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bc41011d0 0x7f2bc41a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:28.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bc4101b70 0x7f2bc41a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:28.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc4107210 0x7f2bc419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:28.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4111320 con 0x7f2bc4107210 2024-03-16T09:40:28.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc41111a0 con 0x7f2bc41011d0 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bcbb7d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc41114a0 con 0x7f2bc4101b70 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc4107210 0x7f2bc419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc4107210 0x7f2bc419aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43216/0 (socket says 172.21.15.77:43216) 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 -- 172.21.15.77:0/3874569841 learned_addr learned my addr 172.21.15.77:0/3874569841 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bc9919700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bc41011d0 0x7f2bc41a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bc9118700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bc4101b70 0x7f2bc41a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 -- 172.21.15.77:0/3874569841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bc4101b70 msgr2=0x7f2bc41a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bc4101b70 0x7f2bc41a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 -- 172.21.15.77:0/3874569841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bc41011d0 msgr2=0x7f2bc41a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bc41011d0 0x7f2bc41a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.131+0000 7f2bca11a700 1 -- 172.21.15.77:0/3874569841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc419b1b0 con 0x7f2bc4107210 2024-03-16T09:40:28.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.132+0000 7f2bca11a700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc4107210 0x7f2bc419aa50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc007ee0 tx=0x7f2bbc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:28.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.132+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc00dd20 con 0x7f2bc4107210 2024-03-16T09:40:28.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.132+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc419b4a0 con 0x7f2bc4107210 2024-03-16T09:40:28.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.132+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bbc004020 con 0x7f2bc4107210 2024-03-16T09:40:28.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.132+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc015660 con 0x7f2bc4107210 2024-03-16T09:40:28.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.132+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc41a7710 con 0x7f2bc4107210 2024-03-16T09:40:28.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.133+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bc406aa60 con 0x7f2bc4107210 2024-03-16T09:40:28.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.134+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2bbc0041d0 con 0x7f2bc4107210 2024-03-16T09:40:28.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bbaffd700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2bb00a2920 0x7f2bb00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:28.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bc9919700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2bb00a2920 0x7f2bb00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:28.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2bbc0cef50 con 0x7f2bc4107210 2024-03-16T09:40:28.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bbaffd700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] conn(0x7f2bb00ac100 0x7f2bb00ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:28.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 --> [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bbc020e50 con 0x7f2bb00ac100 2024-03-16T09:40:28.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f2bbc0cf430 con 0x7f2bc4107210 2024-03-16T09:40:28.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bc9919700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2bb00a2920 0x7f2bb00a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4100c90 tx=0x7f2bc000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:28.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bc9118700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] conn(0x7f2bb00ac100 0x7f2bb00ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:28.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.135+0000 7f2bc9118700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] conn(0x7f2bb00ac100 0x7f2bb00ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:28.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.136+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== osd.6 v2:172.21.15.77:6818/2647291293 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2bbc020e50 con 0x7f2bb00ac100 2024-03-16T09:40:28.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.152+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 --> [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bc41028c0 con 0x7f2bb00ac100 2024-03-16T09:40:28.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.152+0000 7f2bbaffd700 1 -- 172.21.15.77:0/3874569841 <== osd.6 v2:172.21.15.77:6818/2647291293 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bc41028c0 con 0x7f2bb00ac100 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] conn(0x7f2bb00ac100 msgr2=0x7f2bb00ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] conn(0x7f2bb00ac100 0x7f2bb00ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2bb00a2920 msgr2=0x7f2bb00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2bb00a2920 0x7f2bb00a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4100c90 tx=0x7f2bc000b410 comp rx=0 tx=0).stop 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc4107210 msgr2=0x7f2bc419aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 --2- 172.21.15.77:0/3874569841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bc4107210 0x7f2bc419aa50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc007ee0 tx=0x7f2bbc00d3b0 comp rx=0 tx=0).stop 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bca11a700 1 -- 172.21.15.77:0/3874569841 reap_dead start 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 shutdown_connections 2024-03-16T09:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 >> 172.21.15.77:0/3874569841 conn(0x7f2bc4078100 msgr2=0x7f2bc406a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:28.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 shutdown_connections 2024-03-16T09:40:28.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:28.153+0000 7f2bcbb7d700 1 -- 172.21.15.77:0/3874569841 wait complete. 2024-03-16T09:40:29.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:28 smithi119 ceph-mon[35632]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T09:40:29.216 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:28 smithi077 ceph-mon[32171]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T09:40:29.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:28 smithi129 ceph-mon[35841]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T09:40:30.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 -- 172.21.15.77:0/515978528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c106580 msgr2=0x7fad2c10e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 --2- 172.21.15.77:0/515978528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c106580 0x7fad2c10e150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fad2400b0d0 tx=0x7fad2400b3e0 comp rx=0 tx=0).stop 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 -- 172.21.15.77:0/515978528 shutdown_connections 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 --2- 172.21.15.77:0/515978528 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c1014a0 0x7fad2c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 --2- 172.21.15.77:0/515978528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c106580 0x7fad2c10e150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 --2- 172.21.15.77:0/515978528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad2c101e50 0x7fad2c105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 -- 172.21.15.77:0/515978528 >> 172.21.15.77:0/515978528 conn(0x7fad2c078300 msgr2=0x7fad2c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 -- 172.21.15.77:0/515978528 shutdown_connections 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.331+0000 7fad346c5700 1 -- 172.21.15.77:0/515978528 wait complete. 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 Processor -- start 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 -- start start 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad2c1014a0 0x7fad2c102f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c101e50 0x7fad2c103480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c106580 0x7fad2c1039c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c111420 con 0x7fad2c101e50 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1112a0 con 0x7fad2c106580 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad346c5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1115a0 con 0x7fad2c1014a0 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad31c60700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c101e50 0x7fad2c103480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad32461700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad2c1014a0 0x7fad2c102f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad31c60700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c101e50 0x7fad2c103480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43238/0 (socket says 172.21.15.77:43238) 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad32c62700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c106580 0x7fad2c1039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.332+0000 7fad31c60700 1 -- 172.21.15.77:0/2827558366 learned_addr learned my addr 172.21.15.77:0/2827558366 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad32c62700 1 -- 172.21.15.77:0/2827558366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad2c1014a0 msgr2=0x7fad2c102f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad32c62700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad2c1014a0 0x7fad2c102f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad32c62700 1 -- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c101e50 msgr2=0x7fad2c103480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad32c62700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c101e50 0x7fad2c103480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad32c62700 1 -- 172.21.15.77:0/2827558366 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c1a6ef0 con 0x7fad2c106580 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad31c60700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2c101e50 0x7fad2c103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad32c62700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c106580 0x7fad2c1039c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad2400b0d0 tx=0x7fad24007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2401f070 con 0x7fad2c106580 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c1a7180 con 0x7fad2c106580 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad24004c50 con 0x7fad2c106580 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2401b690 con 0x7fad2c106580 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.333+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c1a75d0 con 0x7fad2c106580 2024-03-16T09:40:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.334+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fad2c06ace0 con 0x7fad2c106580 2024-03-16T09:40:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.335+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fad24022070 con 0x7fad2c106580 2024-03-16T09:40:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad237fe700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad180a2920 0x7fad180a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad32461700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad180a2920 0x7fad180a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fad2400f020 con 0x7fad2c106580 2024-03-16T09:40:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad237fe700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] conn(0x7fad180ac100 0x7fad180ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 --> [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad180aebd0 con 0x7fad180ac100 2024-03-16T09:40:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fad240d0140 con 0x7fad2c106580 2024-03-16T09:40:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad31c60700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] conn(0x7fad180ac100 0x7fad180ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.336+0000 7fad32461700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad180a2920 0x7fad180a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fad28009c60 tx=0x7fad28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.337+0000 7fad31c60700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] conn(0x7fad180ac100 0x7fad180ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.337+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== osd.7 v2:172.21.15.119:6816/2297351860 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fad180aebd0 con 0x7fad180ac100 2024-03-16T09:40:30.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.353+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 --> [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fad2c102980 con 0x7fad180ac100 2024-03-16T09:40:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad237fe700 1 -- 172.21.15.77:0/2827558366 <== osd.7 v2:172.21.15.119:6816/2297351860 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fad2c102980 con 0x7fad180ac100 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] conn(0x7fad180ac100 msgr2=0x7fad180ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] conn(0x7fad180ac100 0x7fad180ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad180a2920 msgr2=0x7fad180a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad180a2920 0x7fad180a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fad28009c60 tx=0x7fad28009260 comp rx=0 tx=0).stop 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c106580 msgr2=0x7fad2c1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 --2- 172.21.15.77:0/2827558366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c106580 0x7fad2c1039c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad2400b0d0 tx=0x7fad24007a00 comp rx=0 tx=0).stop 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad32c62700 1 -- 172.21.15.77:0/2827558366 reap_dead start 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 shutdown_connections 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 >> 172.21.15.77:0/2827558366 conn(0x7fad2c078300 msgr2=0x7fad2c105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 shutdown_connections 2024-03-16T09:40:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.354+0000 7fad346c5700 1 -- 172.21.15.77:0/2827558366 wait complete. 2024-03-16T09:40:31.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:30 smithi077 ceph-mon[32171]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-16T09:40:31.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:30 smithi119 ceph-mon[35632]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-16T09:40:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:30 smithi129 ceph-mon[35841]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-16T09:40:31.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.480+0000 7fb7f8068700 1 -- 172.21.15.77:0/1666939422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 msgr2=0x7fb7f010d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:31.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.480+0000 7fb7f8068700 1 --2- 172.21.15.77:0/1666939422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 0x7fb7f010d810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000c210 tx=0x7fb7e000c520 comp rx=0 tx=0).stop 2024-03-16T09:40:31.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.480+0000 7fb7f8068700 1 -- 172.21.15.77:0/1666939422 shutdown_connections 2024-03-16T09:40:31.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.480+0000 7fb7f8068700 1 --2- 172.21.15.77:0/1666939422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7f01069f0 0x7fb7f0106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:31.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.480+0000 7fb7f8068700 1 --2- 172.21.15.77:0/1666939422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 0x7fb7f010d810 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.480+0000 7fb7f8068700 1 --2- 172.21.15.77:0/1666939422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0100c10 0x7fb7f0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.480+0000 7fb7f8068700 1 -- 172.21.15.77:0/1666939422 >> 172.21.15.77:0/1666939422 conn(0x7fb7f0076510 msgr2=0x7fb7f0076920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.481+0000 7fb7f8068700 1 -- 172.21.15.77:0/1666939422 shutdown_connections 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.481+0000 7fb7f8068700 1 -- 172.21.15.77:0/1666939422 wait complete. 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.481+0000 7fb7f8068700 1 Processor -- start 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.481+0000 7fb7f8068700 1 -- start start 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7f0100c10 0x7fb7f019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 0x7fb7f019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f01069f0 0x7fb7f0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f01102a0 con 0x7fb7f01015d0 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0110120 con 0x7fb7f0100c10 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0110420 con 0x7fb7f01069f0 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 0x7fb7f019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 0x7fb7f019eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43256/0 (socket says 172.21.15.77:43256) 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 -- 172.21.15.77:0/997948039 learned_addr learned my addr 172.21.15.77:0/997948039 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5e04700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7f0100c10 0x7fb7f019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f6605700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f01069f0 0x7fb7f0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 -- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f01069f0 msgr2=0x7fb7f0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f01069f0 0x7fb7f0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 -- 172.21.15.77:0/997948039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7f0100c10 msgr2=0x7fb7f019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7f0100c10 0x7fb7f019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 -- 172.21.15.77:0/997948039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f0198f10 con 0x7fb7f01015d0 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f5603700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 0x7fb7f019eaa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400c970 tx=0x7fb7e400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40078d0 con 0x7fb7f01015d0 2024-03-16T09:40:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f0199200 con 0x7fb7f01015d0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7e4018480 con 0x7fb7f01015d0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400f6b0 con 0x7fb7f01015d0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.482+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f01a54e0 con 0x7fb7f01015d0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.483+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb7f00ff560 con 0x7fb7f01015d0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.485+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb7e4018630 con 0x7fb7f01015d0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.485+0000 7fb7eeffd700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb7d80a2920 0x7fb7d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.485+0000 7fb7f5e04700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb7d80a2920 0x7fb7d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.485+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb7e40cf150 con 0x7fb7f01015d0 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.486+0000 7fb7eeffd700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] conn(0x7fb7d80ac100 0x7fb7d80ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.486+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 --> [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7e401ee70 con 0x7fb7d80ac100 2024-03-16T09:40:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.486+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fb7e40cf630 con 0x7fb7f01015d0 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.486+0000 7fb7f5e04700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb7d80a2920 0x7fb7d80a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800ac50 tx=0x7fb7e800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.486+0000 7fb7f6605700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] conn(0x7fb7d80ac100 0x7fb7d80ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.486+0000 7fb7f6605700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] conn(0x7fb7d80ac100 0x7fb7d80ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.487+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== osd.5 v2:172.21.15.129:6808/2429107441 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb7e401ee70 con 0x7fb7d80ac100 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 --> [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb7f0102080 con 0x7fb7d80ac100 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7eeffd700 1 -- 172.21.15.77:0/997948039 <== osd.5 v2:172.21.15.129:6808/2429107441 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb7f0102080 con 0x7fb7d80ac100 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] conn(0x7fb7d80ac100 msgr2=0x7fb7d80ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] conn(0x7fb7d80ac100 0x7fb7d80ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb7d80a2920 msgr2=0x7fb7d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb7d80a2920 0x7fb7d80a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800ac50 tx=0x7fb7e800a380 comp rx=0 tx=0).stop 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 msgr2=0x7fb7f019eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 --2- 172.21.15.77:0/997948039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7f01015d0 0x7fb7f019eaa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400c970 tx=0x7fb7e400cc80 comp rx=0 tx=0).stop 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f6605700 1 -- 172.21.15.77:0/997948039 reap_dead start 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.503+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 shutdown_connections 2024-03-16T09:40:31.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.504+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 >> 172.21.15.77:0/997948039 conn(0x7fb7f0076510 msgr2=0x7fb7f00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:31.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.504+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 shutdown_connections 2024-03-16T09:40:31.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:30.504+0000 7fb7f8068700 1 -- 172.21.15.77:0/997948039 wait complete. 2024-03-16T09:40:32.174 INFO:teuthology.orchestra.run.smithi077.stdout:55834574917 2024-03-16T09:40:32.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 -- 172.21.15.77:0/1084599078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 msgr2=0x7f38f806a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 --2- 172.21.15.77:0/1084599078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 0x7f38f806a3e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f38e80099c0 tx=0x7f38e8009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 -- 172.21.15.77:0/1084599078 shutdown_connections 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 --2- 172.21.15.77:0/1084599078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8108320 0x7f38f8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 --2- 172.21.15.77:0/1084599078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f806a920 0x7f38f810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 --2- 172.21.15.77:0/1084599078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 0x7f38f806a3e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 -- 172.21.15.77:0/1084599078 >> 172.21.15.77:0/1084599078 conn(0x7f38f8078100 msgr2=0x7f38f8078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 -- 172.21.15.77:0/1084599078 shutdown_connections 2024-03-16T09:40:32.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 -- 172.21.15.77:0/1084599078 wait complete. 2024-03-16T09:40:32.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 Processor -- start 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.465+0000 7f38fe4de700 1 -- start start 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38fe4de700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 0x7f38f81012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38fe4de700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f806a920 0x7f38f81017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38fe4de700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8108320 0x7f38f8101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38fe4de700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8112430 con 0x7f38f8069f80 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38fe4de700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f38f81122b0 con 0x7f38f8108320 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38fe4de700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f38f81125b0 con 0x7f38f806a920 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 0x7f38f81012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 0x7f38f81012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43266/0 (socket says 172.21.15.77:43266) 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f7fff700 1 -- 172.21.15.77:0/3912735955 learned_addr learned my addr 172.21.15.77:0/3912735955 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f77fe700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f806a920 0x7f38f81017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:32.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38fca7b700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8108320 0x7f38f8101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f77fe700 1 -- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8108320 msgr2=0x7f38f8101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f77fe700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8108320 0x7f38f8101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f77fe700 1 -- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 msgr2=0x7f38f81012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f77fe700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 0x7f38f81012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f77fe700 1 -- 172.21.15.77:0/3912735955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f81a6ec0 con 0x7f38f806a920 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.466+0000 7f38f7fff700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8069f80 0x7f38f81012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.467+0000 7f38f77fe700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f806a920 0x7f38f81017e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f38e80098e0 tx=0x7f38e8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.467+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8026070 con 0x7f38f806a920 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.467+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f81a7150 con 0x7f38f806a920 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.467+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38e8003c70 con 0x7f38f806a920 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.467+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e800f9c0 con 0x7f38f806a920 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.467+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f81a7600 con 0x7f38f806a920 2024-03-16T09:40:32.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.467+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f38f80629e0 con 0x7f38f806a920 2024-03-16T09:40:32.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.469+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f38e8029070 con 0x7f38f806a920 2024-03-16T09:40:32.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38f57fa700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f38e00a29a0 0x7f38e00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38f7fff700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f38e00a29a0 0x7f38e00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f38e80cf660 con 0x7f38f806a920 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38f57fa700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] conn(0x7f38e00ac050 0x7f38e00ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 --> [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38e00aeb20 con 0x7f38e00ac050 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f38e80cfa50 con 0x7f38f806a920 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38fca7b700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] conn(0x7f38e00ac050 0x7f38e00ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38f7fff700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f38e00a29a0 0x7f38e00a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011550 tx=0x7f38ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.470+0000 7f38fca7b700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] conn(0x7f38e00ac050 0x7f38e00ae470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:32.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.471+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== osd.4 v2:172.21.15.119:6808/3108480529 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38e00aeb20 con 0x7f38e00ac050 2024-03-16T09:40:32.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.487+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 --> [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f38f8062de0 con 0x7f38e00ac050 2024-03-16T09:40:32.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38f57fa700 1 -- 172.21.15.77:0/3912735955 <== osd.4 v2:172.21.15.119:6808/3108480529 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f38f8062de0 con 0x7f38e00ac050 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] conn(0x7f38e00ac050 msgr2=0x7f38e00ae470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] conn(0x7f38e00ac050 0x7f38e00ae470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f38e00a29a0 msgr2=0x7f38e00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f38e00a29a0 0x7f38e00a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011550 tx=0x7f38ec009260 comp rx=0 tx=0).stop 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f806a920 msgr2=0x7f38f81017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 --2- 172.21.15.77:0/3912735955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f806a920 0x7f38f81017e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f38e80098e0 tx=0x7f38e8004270 comp rx=0 tx=0).stop 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fca7b700 1 -- 172.21.15.77:0/3912735955 reap_dead start 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 shutdown_connections 2024-03-16T09:40:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 >> 172.21.15.77:0/3912735955 conn(0x7f38f8078100 msgr2=0x7f38f80fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 shutdown_connections 2024-03-16T09:40:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:32.488+0000 7f38fe4de700 1 -- 172.21.15.77:0/3912735955 wait complete. 2024-03-16T09:40:33.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:32 smithi119 ceph-mon[35632]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:40:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:32 smithi077 ceph-mon[32171]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:40:33.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:32 smithi129 ceph-mon[35841]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:40:33.356 INFO:teuthology.orchestra.run.smithi077.stdout:141733920824 2024-03-16T09:40:33.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-16T09:40:33.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 -- 172.21.15.77:0/248881073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 msgr2=0x7fb68c10f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 --2- 172.21.15.77:0/248881073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 0x7fb68c10f180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb68400b0a0 tx=0x7fb68400b3b0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 -- 172.21.15.77:0/248881073 shutdown_connections 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 --2- 172.21.15.77:0/248881073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68c1083d0 0x7fb68c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 --2- 172.21.15.77:0/248881073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 0x7fb68c10f180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 --2- 172.21.15.77:0/248881073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c102320 0x7fb68c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 -- 172.21.15.77:0/248881073 >> 172.21.15.77:0/248881073 conn(0x7fb68c079400 msgr2=0x7fb68c079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.569+0000 7fb6917ee700 1 -- 172.21.15.77:0/248881073 shutdown_connections 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.570+0000 7fb6917ee700 1 -- 172.21.15.77:0/248881073 wait complete. 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.570+0000 7fb6917ee700 1 Processor -- start 2024-03-16T09:40:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.570+0000 7fb6917ee700 1 -- start start 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c102320 0x7fb68c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 0x7fb68c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68c1083d0 0x7fb68c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c1124c0 con 0x7fb68c102ce0 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c112340 con 0x7fb68c1083d0 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c112640 con 0x7fb68c102320 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 0x7fb68c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 0x7fb68c19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43270/0 (socket says 172.21.15.77:43270) 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 -- 172.21.15.77:0/3429113201 learned_addr learned my addr 172.21.15.77:0/3429113201 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb68affd700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c102320 0x7fb68c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb68b7fe700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68c1083d0 0x7fb68c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 -- 172.21.15.77:0/3429113201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c102320 msgr2=0x7fb68c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c102320 0x7fb68c19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 -- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68c1083d0 msgr2=0x7fb68c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68c1083d0 0x7fb68c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 -- 172.21.15.77:0/3429113201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68c199020 con 0x7fb68c102ce0 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6827fc700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 0x7fb68c19ebb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb67400e9c0 tx=0x7fb67400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67400cbd0 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb67400ee60 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb68c199310 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6740055b0 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.571+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb68c1a55f0 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.572+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb68c062960 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.573+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb674004750 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.574+0000 7fb688ff9700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb6780a29a0 0x7fb6780a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.574+0000 7fb68affd700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb6780a29a0 0x7fb6780a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.574+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb6740cec40 con 0x7fb68c102ce0 2024-03-16T09:40:33.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.574+0000 7fb688ff9700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] conn(0x7fb6780ac180 0x7fb6780ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:33.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.574+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 --> [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6780aec50 con 0x7fb6780ac180 2024-03-16T09:40:33.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.574+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fb6740d1390 con 0x7fb68c102ce0 2024-03-16T09:40:33.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.574+0000 7fb68b7fe700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] conn(0x7fb6780ac180 0x7fb6780ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:33.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.575+0000 7fb68affd700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb6780a29a0 0x7fb6780a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb67c007be0 tx=0x7fb67c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:33.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.575+0000 7fb68b7fe700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] conn(0x7fb6780ac180 0x7fb6780ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:33.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.576+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== osd.10 v2:172.21.15.119:6824/1354177519 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb6780aec50 con 0x7fb6780ac180 2024-03-16T09:40:33.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 --> [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb68c062e10 con 0x7fb6780ac180 2024-03-16T09:40:33.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb688ff9700 1 -- 172.21.15.77:0/3429113201 <== osd.10 v2:172.21.15.119:6824/1354177519 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb68c062e10 con 0x7fb6780ac180 2024-03-16T09:40:33.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] conn(0x7fb6780ac180 msgr2=0x7fb6780ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:33.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb6917ee700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] conn(0x7fb6780ac180 0x7fb6780ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb6780a29a0 msgr2=0x7fb6780a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:33.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb6917ee700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb6780a29a0 0x7fb6780a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb67c007be0 tx=0x7fb67c0073d0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 msgr2=0x7fb68c19ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb6917ee700 1 --2- 172.21.15.77:0/3429113201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb68c102ce0 0x7fb68c19ebb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb67400e9c0 tx=0x7fb67400ecd0 comp rx=0 tx=0).stop 2024-03-16T09:40:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.593+0000 7fb68b7fe700 1 -- 172.21.15.77:0/3429113201 reap_dead start 2024-03-16T09:40:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.594+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 shutdown_connections 2024-03-16T09:40:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.594+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 >> 172.21.15.77:0/3429113201 conn(0x7fb68c079400 msgr2=0x7fb68c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.594+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 shutdown_connections 2024-03-16T09:40:33.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:33.594+0000 7fb6917ee700 1 -- 172.21.15.77:0/3429113201 wait complete. 2024-03-16T09:40:34.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 -- 172.21.15.77:0/2373402380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834108320 msgr2=0x7fb834108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 --2- 172.21.15.77:0/2373402380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834108320 0x7fb834108700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb8240099c0 tx=0x7fb824009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 -- 172.21.15.77:0/2373402380 shutdown_connections 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 --2- 172.21.15.77:0/2373402380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb83406a920 0x7fb83410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 --2- 172.21.15.77:0/2373402380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834108320 0x7fb834108700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 --2- 172.21.15.77:0/2373402380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb834069f80 0x7fb83406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 -- 172.21.15.77:0/2373402380 >> 172.21.15.77:0/2373402380 conn(0x7fb834078100 msgr2=0x7fb834078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 -- 172.21.15.77:0/2373402380 shutdown_connections 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.550+0000 7fb8394cd700 1 -- 172.21.15.77:0/2373402380 wait complete. 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 Processor -- start 2024-03-16T09:40:34.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 -- start start 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834069f80 0x7fb83419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb83406a920 0x7fb83419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb834108320 0x7fb834198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb834112410 con 0x7fb834069f80 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb834112290 con 0x7fb83406a920 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8394cd700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb834112590 con 0x7fb834108320 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb832ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834069f80 0x7fb83419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb82a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb83406a920 0x7fb83419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb832ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834069f80 0x7fb83419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43276/0 (socket says 172.21.15.77:43276) 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb832ffd700 1 -- 172.21.15.77:0/1608726024 learned_addr learned my addr 172.21.15.77:0/1608726024 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:34.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.551+0000 7fb8337fe700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb834108320 0x7fb834198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb832ffd700 1 -- 172.21.15.77:0/1608726024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb834108320 msgr2=0x7fb834198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb832ffd700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb834108320 0x7fb834198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb832ffd700 1 -- 172.21.15.77:0/1608726024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb83406a920 msgr2=0x7fb83419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb832ffd700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb83406a920 0x7fb83419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb832ffd700 1 -- 172.21.15.77:0/1608726024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb834199030 con 0x7fb834069f80 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb832ffd700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834069f80 0x7fb83419e680 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb824006010 tx=0x7fb824009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824026070 con 0x7fb834069f80 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8341992c0 con 0x7fb834069f80 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb824004ca0 con 0x7fb834069f80 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824003d20 con 0x7fb834069f80 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.552+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8341a5600 con 0x7fb834069f80 2024-03-16T09:40:34.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.553+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb834062a20 con 0x7fb834069f80 2024-03-16T09:40:34.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.554+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb824029070 con 0x7fb834069f80 2024-03-16T09:40:34.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.554+0000 7fb830ff9700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb8200a2920 0x7fb8200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb82a7fc700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb8200a2920 0x7fb8200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb8240cf6e0 con 0x7fb834069f80 2024-03-16T09:40:34.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb830ff9700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] conn(0x7fb8200ac100 0x7fb8200ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 --> [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb824017e50 con 0x7fb8200ac100 2024-03-16T09:40:34.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fb8240cfbc0 con 0x7fb834069f80 2024-03-16T09:40:34.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb8337fe700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] conn(0x7fb8200ac100 0x7fb8200ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb82a7fc700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb8200a2920 0x7fb8200a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb834199f10 tx=0x7fb81c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:34.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.555+0000 7fb8337fe700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] conn(0x7fb8200ac100 0x7fb8200ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:34.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.556+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== osd.9 v2:172.21.15.77:6826/1881713362 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb824017e50 con 0x7fb8200ac100 2024-03-16T09:40:34.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.573+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 --> [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb834062e20 con 0x7fb8200ac100 2024-03-16T09:40:34.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb830ff9700 1 -- 172.21.15.77:0/1608726024 <== osd.9 v2:172.21.15.77:6826/1881713362 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb834062e20 con 0x7fb8200ac100 2024-03-16T09:40:34.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] conn(0x7fb8200ac100 msgr2=0x7fb8200ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] conn(0x7fb8200ac100 0x7fb8200ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb8200a2920 msgr2=0x7fb8200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb8200a2920 0x7fb8200a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb834199f10 tx=0x7fb81c0073d0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834069f80 msgr2=0x7fb83419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 --2- 172.21.15.77:0/1608726024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb834069f80 0x7fb83419e680 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb824006010 tx=0x7fb824009f80 comp rx=0 tx=0).stop 2024-03-16T09:40:34.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8337fe700 1 -- 172.21.15.77:0/1608726024 reap_dead start 2024-03-16T09:40:34.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 shutdown_connections 2024-03-16T09:40:34.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 >> 172.21.15.77:0/1608726024 conn(0x7fb834078100 msgr2=0x7fb8340ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:34.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 shutdown_connections 2024-03-16T09:40:34.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.574+0000 7fb8394cd700 1 -- 172.21.15.77:0/1608726024 wait complete. 2024-03-16T09:40:34.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 -- 172.21.15.77:0/2076116728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290069f80 msgr2=0x7fd29006a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 --2- 172.21.15.77:0/2076116728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290069f80 0x7fd29006a3e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0099c0 tx=0x7fd27c009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 -- 172.21.15.77:0/2076116728 shutdown_connections 2024-03-16T09:40:34.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 --2- 172.21.15.77:0/2076116728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29006a920 0x7fd29010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 --2- 172.21.15.77:0/2076116728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd290108320 0x7fd290108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 --2- 172.21.15.77:0/2076116728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290069f80 0x7fd29006a3e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 -- 172.21.15.77:0/2076116728 >> 172.21.15.77:0/2076116728 conn(0x7fd290078100 msgr2=0x7fd290078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:34.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 -- 172.21.15.77:0/2076116728 shutdown_connections 2024-03-16T09:40:34.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.778+0000 7fd29723a700 1 -- 172.21.15.77:0/2076116728 wait complete. 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 Processor -- start 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 -- start start 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd290069f80 0x7fd29019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29006a920 0x7fd29019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290108320 0x7fd290198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd290112430 con 0x7fd290108320 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901122b0 con 0x7fd29006a920 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd29723a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901125b0 con 0x7fd290069f80 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd2957d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290108320 0x7fd290198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd2957d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290108320 0x7fd290198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43302/0 (socket says 172.21.15.77:43302) 2024-03-16T09:40:34.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd2957d7700 1 -- 172.21.15.77:0/3275231521 learned_addr learned my addr 172.21.15.77:0/3275231521 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd294fd6700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd290069f80 0x7fd29019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29006a920 0x7fd29019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.779+0000 7fd2957d7700 1 -- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd290069f80 msgr2=0x7fd29019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd2957d7700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd290069f80 0x7fd29019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd2957d7700 1 -- 172.21.15.77:0/3275231521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29006a920 msgr2=0x7fd29019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd2957d7700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29006a920 0x7fd29019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd2957d7700 1 -- 172.21.15.77:0/3275231521 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd290199050 con 0x7fd290108320 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29006a920 0x7fd29019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd294fd6700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd290069f80 0x7fd29019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd2957d7700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290108320 0x7fd290198770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd28800ef30 tx=0x7fd28800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2880092e0 con 0x7fd290108320 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd288004bb0 con 0x7fd290108320 2024-03-16T09:40:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2901992e0 con 0x7fd290108320 2024-03-16T09:40:34.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288004020 con 0x7fd290108320 2024-03-16T09:40:34.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.780+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2901a5620 con 0x7fd290108320 2024-03-16T09:40:34.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.781+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd290199620 con 0x7fd290108320 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.782+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd28801b030 con 0x7fd290108320 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.782+0000 7fd28dffb700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2800a2920 0x7fd2800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd294fd6700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2800a2920 0x7fd2800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd2880cee60 con 0x7fd290108320 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd28dffb700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] conn(0x7fd2800ac100 0x7fd2800ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 --> [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2800aebd0 con 0x7fd2800ac100 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fd2880cf340 con 0x7fd290108320 2024-03-16T09:40:34.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd294fd6700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2800a2920 0x7fd2800a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd284009fd0 tx=0x7fd284009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] conn(0x7fd2800ac100 0x7fd2800ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.783+0000 7fd28ffff700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] conn(0x7fd2800ac100 0x7fd2800ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.784+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== osd.11 v2:172.21.15.129:6824/913848880 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2800aebd0 con 0x7fd2800ac100 2024-03-16T09:40:34.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.800+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 --> [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd290062cb0 con 0x7fd2800ac100 2024-03-16T09:40:34.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.800+0000 7fd28dffb700 1 -- 172.21.15.77:0/3275231521 <== osd.11 v2:172.21.15.129:6824/913848880 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd290062cb0 con 0x7fd2800ac100 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] conn(0x7fd2800ac100 msgr2=0x7fd2800ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] conn(0x7fd2800ac100 0x7fd2800ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2800a2920 msgr2=0x7fd2800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd2800a2920 0x7fd2800a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd284009fd0 tx=0x7fd284009260 comp rx=0 tx=0).stop 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290108320 msgr2=0x7fd290198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 --2- 172.21.15.77:0/3275231521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd290108320 0x7fd290198770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd28800ef30 tx=0x7fd28800c3a0 comp rx=0 tx=0).stop 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd2957d7700 1 -- 172.21.15.77:0/3275231521 reap_dead start 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 shutdown_connections 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 >> 172.21.15.77:0/3275231521 conn(0x7fd290078100 msgr2=0x7fd2900ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:34.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 shutdown_connections 2024-03-16T09:40:34.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:34.801+0000 7fd29723a700 1 -- 172.21.15.77:0/3275231521 wait complete. 2024-03-16T09:40:35.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:34 smithi077 ceph-mon[32171]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:35.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:34 smithi119 ceph-mon[35632]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:35.201 INFO:teuthology.orchestra.run.smithi077.stdout:223338299432 2024-03-16T09:40:35.202 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-16T09:40:35.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:34 smithi129 ceph-mon[35841]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:36.317 INFO:teuthology.orchestra.run.smithi077.stdout:313532612633 2024-03-16T09:40:36.317 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-16T09:40:36.717 INFO:teuthology.orchestra.run.smithi077.stdout:201863462955 2024-03-16T09:40:36.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-16T09:40:37.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:36 smithi077 ceph-mon[32171]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:37.140 INFO:teuthology.orchestra.run.smithi077.stdout:167503724594 2024-03-16T09:40:37.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-16T09:40:37.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:36 smithi119 ceph-mon[35632]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:37.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:36 smithi129 ceph-mon[35841]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:37.740 INFO:teuthology.orchestra.run.smithi077.stdout:347892350997 2024-03-16T09:40:37.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-16T09:40:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 -- 172.21.15.77:0/815034161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101460 msgr2=0x7f7f34101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 --2- 172.21.15.77:0/815034161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101460 0x7f7f34101840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7f240099c0 tx=0x7f7f24009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 -- 172.21.15.77:0/815034161 shutdown_connections 2024-03-16T09:40:38.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 --2- 172.21.15.77:0/815034161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f341064b0 0x7f7f3410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 --2- 172.21.15.77:0/815034161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101460 0x7f7f34101840 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 --2- 172.21.15.77:0/815034161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f34101e10 0x7f7f34105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 -- 172.21.15.77:0/815034161 >> 172.21.15.77:0/815034161 conn(0x7f7f34077220 msgr2=0x7f7f34077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:38.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 -- 172.21.15.77:0/815034161 shutdown_connections 2024-03-16T09:40:38.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.301+0000 7f7f38d45700 1 -- 172.21.15.77:0/815034161 wait complete. 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 Processor -- start 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 -- start start 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f34101460 0x7f7f3419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101e10 0x7f7f3419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f341064b0 0x7f7f341987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7f34111360 con 0x7f7f34101e10 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341111e0 con 0x7f7f341064b0 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f38d45700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341114e0 con 0x7f7f34101460 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f31d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101e10 0x7f7f3419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f31d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101e10 0x7f7f3419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46790/0 (socket says 172.21.15.77:46790) 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f31d9b700 1 -- 172.21.15.77:0/2767530157 learned_addr learned my addr 172.21.15.77:0/2767530157 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f3259c700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f34101460 0x7f7f3419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.302+0000 7f7f32d9d700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f341064b0 0x7f7f341987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f3259c700 1 -- 172.21.15.77:0/2767530157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f341064b0 msgr2=0x7f7f341987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f3259c700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f341064b0 0x7f7f341987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f3259c700 1 -- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101e10 msgr2=0x7f7f3419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f3259c700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101e10 0x7f7f3419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f3259c700 1 -- 172.21.15.77:0/2767530157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f34198fc0 con 0x7f7f34101460 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f32d9d700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f341064b0 0x7f7f341987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f31d9b700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f34101e10 0x7f7f3419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f3259c700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f34101460 0x7f7f3419e610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7f240098e0 tx=0x7f7f24004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24026070 con 0x7f7f34101460 2024-03-16T09:40:38.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f24003c70 con 0x7f7f34101460 2024-03-16T09:40:38.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f34199250 con 0x7f7f34101460 2024-03-16T09:40:38.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2400f950 con 0x7f7f34101460 2024-03-16T09:40:38.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.303+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f34102da0 con 0x7f7f34101460 2024-03-16T09:40:38.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.304+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f34062a60 con 0x7f7f34101460 2024-03-16T09:40:38.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.305+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7f24029070 con 0x7f7f34101460 2024-03-16T09:40:38.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.305+0000 7f7f237fe700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7f1c0a29a0 0x7f7f1c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f31d9b700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7f1c0a29a0 0x7f7f1c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7f240cf130 con 0x7f7f34101460 2024-03-16T09:40:38.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f237fe700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] conn(0x7f7f1c0ac180 0x7f7f1c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 --> [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f1c0aec50 con 0x7f7f1c0ac180 2024-03-16T09:40:38.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f7f240cf590 con 0x7f7f34101460 2024-03-16T09:40:38.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f32d9d700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] conn(0x7f7f1c0ac180 0x7f7f1c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f31d9b700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7f1c0a29a0 0x7f7f1c0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f34100f80 tx=0x7f7f28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:38.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.306+0000 7f7f32d9d700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] conn(0x7f7f1c0ac180 0x7f7f1c0ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:38.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.307+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== osd.3 v2:172.21.15.77:6810/698693547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f1c0aec50 con 0x7f7f1c0ac180 2024-03-16T09:40:38.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.323+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 --> [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f34062e60 con 0x7f7f1c0ac180 2024-03-16T09:40:38.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.323+0000 7f7f237fe700 1 -- 172.21.15.77:0/2767530157 <== osd.3 v2:172.21.15.77:6810/698693547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7f34062e60 con 0x7f7f1c0ac180 2024-03-16T09:40:38.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] conn(0x7f7f1c0ac180 msgr2=0x7f7f1c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] conn(0x7f7f1c0ac180 0x7f7f1c0ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7f1c0a29a0 msgr2=0x7f7f1c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7f1c0a29a0 0x7f7f1c0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f34100f80 tx=0x7f7f28009260 comp rx=0 tx=0).stop 2024-03-16T09:40:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f34101460 msgr2=0x7f7f3419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 --2- 172.21.15.77:0/2767530157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f34101460 0x7f7f3419e610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7f240098e0 tx=0x7f7f24004270 comp rx=0 tx=0).stop 2024-03-16T09:40:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f32d9d700 1 -- 172.21.15.77:0/2767530157 reap_dead start 2024-03-16T09:40:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 shutdown_connections 2024-03-16T09:40:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 >> 172.21.15.77:0/2767530157 conn(0x7f7f34077220 msgr2=0x7f7f34105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 shutdown_connections 2024-03-16T09:40:38.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.324+0000 7f7f38d45700 1 -- 172.21.15.77:0/2767530157 wait complete. 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 -- 172.21.15.77:0/2182754313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1aa0108300 msgr2=0x7f1aa01086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 --2- 172.21.15.77:0/2182754313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1aa0108300 0x7f1aa01086e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009920 tx=0x7f1a9c009c30 comp rx=0 tx=0).stop 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 -- 172.21.15.77:0/2182754313 shutdown_connections 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 --2- 172.21.15.77:0/2182754313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1aa0108300 0x7f1aa01086e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 --2- 172.21.15.77:0/2182754313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa0102ce0 0x7f1aa010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 --2- 172.21.15.77:0/2182754313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0102320 0x7f1aa01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 -- 172.21.15.77:0/2182754313 >> 172.21.15.77:0/2182754313 conn(0x7f1aa0079400 msgr2=0x7f1aa0079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 -- 172.21.15.77:0/2182754313 shutdown_connections 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.691+0000 7f1aa7360700 1 -- 172.21.15.77:0/2182754313 wait complete. 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 Processor -- start 2024-03-16T09:40:38.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 -- start start 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0102320 0x7f1aa019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa0102ce0 0x7f1aa019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1aa0108300 0x7f1aa01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa0112430 con 0x7f1aa0102ce0 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa01122b0 con 0x7f1aa0108300 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa7360700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa01125b0 con 0x7f1aa0102320 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa48fb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa0102ce0 0x7f1aa019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa48fb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa0102ce0 0x7f1aa019eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46806/0 (socket says 172.21.15.77:46806) 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa48fb700 1 -- 172.21.15.77:0/2059082387 learned_addr learned my addr 172.21.15.77:0/2059082387 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa58fd700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1aa0108300 0x7f1aa01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.692+0000 7f1aa50fc700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0102320 0x7f1aa019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa48fb700 1 -- 172.21.15.77:0/2059082387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0102320 msgr2=0x7f1aa019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa48fb700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0102320 0x7f1aa019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa48fb700 1 -- 172.21.15.77:0/2059082387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1aa0108300 msgr2=0x7f1aa01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa48fb700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1aa0108300 0x7f1aa01987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa48fb700 1 -- 172.21.15.77:0/2059082387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa0198f90 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa48fb700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa0102ce0 0x7f1aa019eb70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000e9a0 tx=0x7f1a9000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9000ca10 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa0199280 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a9000cbb0 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a90018680 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.693+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa01a5490 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.694+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1aa0062a60 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.695+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1a90010430 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.695+0000 7f1a967fc700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a8c0a2920 0x7f1a8c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1aa50fc700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a8c0a2920 0x7f1a8c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1a900cf050 con 0x7f1aa0102ce0 2024-03-16T09:40:38.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1a967fc700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] conn(0x7f1a8c0ac100 0x7f1a8c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 --> [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a9001bec0 con 0x7f1a8c0ac100 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f1a900cf530 con 0x7f1aa0102ce0 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1aa50fc700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a8c0a2920 0x7f1a8c0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009680 tx=0x7f1a9c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1aa58fd700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] conn(0x7f1a8c0ac100 0x7f1a8c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.696+0000 7f1aa58fd700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] conn(0x7f1a8c0ac100 0x7f1a8c0ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.697+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== osd.0 v2:172.21.15.77:6802/3264923905 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a9001bec0 con 0x7f1a8c0ac100 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 --> [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1aa0062ee0 con 0x7f1a8c0ac100 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1a967fc700 1 -- 172.21.15.77:0/2059082387 <== osd.0 v2:172.21.15.77:6802/3264923905 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1aa0062ee0 con 0x7f1a8c0ac100 2024-03-16T09:40:38.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] conn(0x7f1a8c0ac100 msgr2=0x7f1a8c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] conn(0x7f1a8c0ac100 0x7f1a8c0ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a8c0a2920 msgr2=0x7f1a8c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1a8c0a2920 0x7f1a8c0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009680 tx=0x7f1a9c019040 comp rx=0 tx=0).stop 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa0102ce0 msgr2=0x7f1aa019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 --2- 172.21.15.77:0/2059082387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa0102ce0 0x7f1aa019eb70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000e9a0 tx=0x7f1a9000ecb0 comp rx=0 tx=0).stop 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa58fd700 1 -- 172.21.15.77:0/2059082387 reap_dead start 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 shutdown_connections 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 >> 172.21.15.77:0/2059082387 conn(0x7f1aa0079400 msgr2=0x7f1aa0101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 shutdown_connections 2024-03-16T09:40:38.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:38.713+0000 7f1aa7360700 1 -- 172.21.15.77:0/2059082387 wait complete. 2024-03-16T09:40:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:38 smithi119 ceph-mon[35632]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:40:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:38 smithi077 ceph-mon[32171]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:40:39.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:38 smithi129 ceph-mon[35841]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:40:39.666 INFO:teuthology.orchestra.run.smithi077.stdout:292057776159 2024-03-16T09:40:39.667 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-16T09:40:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:34359738444 2024-03-16T09:40:40.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-16T09:40:40.540 INFO:teuthology.orchestra.run.smithi077.stdout:107374182461 2024-03-16T09:40:40.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-16T09:40:40.568 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:40.569 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:40.575 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:40 smithi119 ceph-mon[35632]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:40:41.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:40 smithi077 ceph-mon[32171]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:40:41.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:40 smithi129 ceph-mon[35841]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:40:43.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:42 smithi077 ceph-mon[32171]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:43.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:42 smithi119 ceph-mon[35632]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:43.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:42 smithi129 ceph-mon[35841]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T09:40:43.999 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:43 smithi077 ceph-mon[32171]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:44.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:43 smithi129 ceph-mon[35841]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:44.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:43 smithi119 ceph-mon[35632]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.334+0000 7f478400d700 1 -- 172.21.15.77:0/4218845588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c108320 msgr2=0x7f477c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.334+0000 7f478400d700 1 --2- 172.21.15.77:0/4218845588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c108320 0x7f477c108700 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f47780099c0 tx=0x7f4778009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- 172.21.15.77:0/4218845588 shutdown_connections 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 --2- 172.21.15.77:0/4218845588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477c06a920 0x7f477c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 --2- 172.21.15.77:0/4218845588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c108320 0x7f477c108700 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 --2- 172.21.15.77:0/4218845588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f477c069f80 0x7f477c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- 172.21.15.77:0/4218845588 >> 172.21.15.77:0/4218845588 conn(0x7f477c078100 msgr2=0x7f477c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- 172.21.15.77:0/4218845588 shutdown_connections 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- 172.21.15.77:0/4218845588 wait complete. 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 Processor -- start 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- start start 2024-03-16T09:40:45.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c069f80 0x7f477c1012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477c06a920 0x7f477c1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f477c108320 0x7f477c101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f477c112460 con 0x7f477c069f80 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1122e0 con 0x7f477c108320 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.335+0000 7f478400d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f477c1125e0 con 0x7f477c06a920 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c069f80 0x7f477c1012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f47825aa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f477c108320 0x7f477c101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c069f80 0x7f477c1012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46828/0 (socket says 172.21.15.77:46828) 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 -- 172.21.15.77:0/3221603135 learned_addr learned my addr 172.21.15.77:0/3221603135 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f47815a8700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477c06a920 0x7f477c1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 -- 172.21.15.77:0/3221603135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477c06a920 msgr2=0x7f477c1017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477c06a920 0x7f477c1017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 -- 172.21.15.77:0/3221603135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f477c108320 msgr2=0x7f477c101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f477c108320 0x7f477c101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 -- 172.21.15.77:0/3221603135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c1a6ec0 con 0x7f477c069f80 2024-03-16T09:40:45.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4781da9700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c069f80 0x7f477c1012a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4778005910 tx=0x7f47780045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:45.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4772ffd700 1 -- 172.21.15.77:0/3221603135 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778026070 con 0x7f477c069f80 2024-03-16T09:40:45.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f477c1a7150 con 0x7f477c069f80 2024-03-16T09:40:45.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4772ffd700 1 -- 172.21.15.77:0/3221603135 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f477800f460 con 0x7f477c069f80 2024-03-16T09:40:45.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.336+0000 7f4772ffd700 1 -- 172.21.15.77:0/3221603135 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778009e60 con 0x7f477c069f80 2024-03-16T09:40:45.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.337+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f477c1a7600 con 0x7f477c069f80 2024-03-16T09:40:45.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.338+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f477c068f90 con 0x7f477c069f80 2024-03-16T09:40:45.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.338+0000 7f4772ffd700 1 -- 172.21.15.77:0/3221603135 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4778029070 con 0x7f477c069f80 2024-03-16T09:40:45.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.339+0000 7f4772ffd700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47680a29a0 0x7f47680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:45.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.339+0000 7f47815a8700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47680a29a0 0x7f47680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:45.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.339+0000 7f4772ffd700 1 -- 172.21.15.77:0/3221603135 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f47780cf140 con 0x7f477c069f80 2024-03-16T09:40:45.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.339+0000 7f47815a8700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47680a29a0 0x7f47680a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f477c0787e0 tx=0x7f476c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:45.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.343+0000 7f4772ffd700 1 -- 172.21.15.77:0/3221603135 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f477805d740 con 0x7f477c069f80 2024-03-16T09:40:45.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.548+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f477c02db70 con 0x7f477c069f80 2024-03-16T09:40:45.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.548+0000 7f4772ffd700 1 -- 172.21.15.77:0/3221603135 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4778013070 con 0x7f477c069f80 2024-03-16T09:40:45.550 INFO:teuthology.orchestra.run.smithi077.stdout:347892350999 2024-03-16T09:40:45.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47680a29a0 msgr2=0x7f47680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:45.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47680a29a0 0x7f47680a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f477c0787e0 tx=0x7f476c0073d0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c069f80 msgr2=0x7f477c1012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:45.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c069f80 0x7f477c1012a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4778005910 tx=0x7f47780045e0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 shutdown_connections 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47680a29a0 0x7f47680a4e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f477c108320 0x7f477c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f477c069f80 0x7f477c1012a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 --2- 172.21.15.77:0/3221603135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f477c06a920 0x7f477c1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 >> 172.21.15.77:0/3221603135 conn(0x7f477c078100 msgr2=0x7f477c100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 shutdown_connections 2024-03-16T09:40:45.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:45.550+0000 7f478400d700 1 -- 172.21.15.77:0/3221603135 wait complete. 2024-03-16T09:40:45.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:45 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3221603135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T09:40:45.901 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:45.901 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:45.902 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:45.902 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:45.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:45 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3221603135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.004+0000 7f5dfc47f700 1 -- 172.21.15.77:0/3913054773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df4101e10 msgr2=0x7f5df4105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.004+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/3913054773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df4101e10 0x7f5df4105dd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5de40099c0 tx=0x7f5de4009cd0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 -- 172.21.15.77:0/3913054773 shutdown_connections 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/3913054773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df41064b0 0x7f5df410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/3913054773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df4101460 0x7f5df4101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/3913054773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df4101e10 0x7f5df4105dd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 -- 172.21.15.77:0/3913054773 >> 172.21.15.77:0/3913054773 conn(0x7f5df4077220 msgr2=0x7f5df4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 -- 172.21.15.77:0/3913054773 shutdown_connections 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 -- 172.21.15.77:0/3913054773 wait complete. 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 Processor -- start 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.005+0000 7f5dfc47f700 1 -- start start 2024-03-16T09:40:46.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df4101460 0x7f5df4199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df4101e10 0x7f5df4199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df41064b0 0x7f5df41a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5df4111340 con 0x7f5df41064b0 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5df41111c0 con 0x7f5df4101460 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5df41114c0 con 0x7f5df4101e10 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df4101460 0x7f5df4199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfaa1c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df41064b0 0x7f5df41a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df4101460 0x7f5df4199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:48422/0 (socket says 172.21.15.77:48422) 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 -- 172.21.15.77:0/1067420329 learned_addr learned my addr 172.21.15.77:0/1067420329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5df9a1a700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df4101e10 0x7f5df4199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 -- 172.21.15.77:0/1067420329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df4101e10 msgr2=0x7f5df4199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df4101e10 0x7f5df4199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 -- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df41064b0 msgr2=0x7f5df41a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df41064b0 0x7f5df41a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 -- 172.21.15.77:0/1067420329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df41a4e00 con 0x7f5df4101460 2024-03-16T09:40:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfa21b700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df4101460 0x7f5df4199840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5df000e9c0 tx=0x7f5df000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5deb7fe700 1 -- 172.21.15.77:0/1067420329 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df000cbd0 con 0x7f5df4101460 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df41a4fd0 con 0x7f5df4101460 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5deb7fe700 1 -- 172.21.15.77:0/1067420329 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5df000ee60 con 0x7f5df4101460 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5deb7fe700 1 -- 172.21.15.77:0/1067420329 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df00055b0 con 0x7f5df4101460 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.006+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df41a53d0 con 0x7f5df4101460 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.007+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df4068f90 con 0x7f5df4101460 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.008+0000 7f5deb7fe700 1 -- 172.21.15.77:0/1067420329 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5df0044940 con 0x7f5df4101460 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.009+0000 7f5deb7fe700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5de00a2920 0x7f5de00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.009+0000 7f5df9a1a700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5de00a2920 0x7f5de00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:46.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.009+0000 7f5deb7fe700 1 -- 172.21.15.77:0/1067420329 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5df00ced90 con 0x7f5df4101460 2024-03-16T09:40:46.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.010+0000 7f5df9a1a700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5de00a2920 0x7f5de00a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5de4000c00 tx=0x7f5de4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:46.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.013+0000 7f5deb7fe700 1 -- 172.21.15.77:0/1067420329 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5df00203a0 con 0x7f5df4101460 2024-03-16T09:40:46.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:45 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3221603135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T09:40:46.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.213+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5df402dc10 con 0x7f5df4101460 2024-03-16T09:40:46.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.213+0000 7f5deb7fe700 1 -- 172.21.15.77:0/1067420329 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5df009bd80 con 0x7f5df4101460 2024-03-16T09:40:46.215 INFO:teuthology.orchestra.run.smithi077.stdout:55834574920 2024-03-16T09:40:46.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5de00a2920 msgr2=0x7f5de00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5de00a2920 0x7f5de00a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5de4000c00 tx=0x7f5de4005fb0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df4101460 msgr2=0x7f5df4199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df4101460 0x7f5df4199840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5df000e9c0 tx=0x7f5df000ecd0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 shutdown_connections 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5de00a2920 0x7f5de00a4de0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df4101460 0x7f5df4199840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5df41064b0 0x7f5df41a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 --2- 172.21.15.77:0/1067420329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df4101e10 0x7f5df4199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 >> 172.21.15.77:0/1067420329 conn(0x7f5df4077220 msgr2=0x7f5df40ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 shutdown_connections 2024-03-16T09:40:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.215+0000 7f5dfc47f700 1 -- 172.21.15.77:0/1067420329 wait complete. 2024-03-16T09:40:46.523 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350997 got 347892350999 for osd.11 2024-03-16T09:40:46.523 DEBUG:teuthology.parallel:result is None 2024-03-16T09:40:46.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:46 smithi077 ceph-mon[32171]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T09:40:46.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:46 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1067420329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 -- 172.21.15.77:0/929624616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28141064e0 msgr2=0x7f281410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 --2- 172.21.15.77:0/929624616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28141064e0 0x7f281410e0b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f280c00b0a0 tx=0x7f280c00b3b0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 -- 172.21.15.77:0/929624616 shutdown_connections 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 --2- 172.21.15.77:0/929624616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2814101420 0x7f2814101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 --2- 172.21.15.77:0/929624616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28141064e0 0x7f281410e0b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 --2- 172.21.15.77:0/929624616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2814101dd0 0x7f2814105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 -- 172.21.15.77:0/929624616 >> 172.21.15.77:0/929624616 conn(0x7f2814078140 msgr2=0x7f2814078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:46.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 -- 172.21.15.77:0/929624616 shutdown_connections 2024-03-16T09:40:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.766+0000 7f281b688700 1 -- 172.21.15.77:0/929624616 wait complete. 2024-03-16T09:40:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 Processor -- start 2024-03-16T09:40:46.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 -- start start 2024-03-16T09:40:46.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2814101420 0x7f2814197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2814101dd0 0x7f2814197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28141064e0 0x7f2814198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2814111370 con 0x7f2814101dd0 2024-03-16T09:40:46.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f28141111f0 con 0x7f28141064e0 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f281b688700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28141114f0 con 0x7f2814101420 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f2819c25700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28141064e0 0x7f2814198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f2819c25700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28141064e0 0x7f2814198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:48440/0 (socket says 172.21.15.77:48440) 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.767+0000 7f2819c25700 1 -- 172.21.15.77:0/644385168 learned_addr learned my addr 172.21.15.77:0/644385168 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2818c23700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2814101dd0 0x7f2814197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2819c25700 1 -- 172.21.15.77:0/644385168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2814101420 msgr2=0x7f2814197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2819c25700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2814101420 0x7f2814197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2819c25700 1 -- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2814101dd0 msgr2=0x7f2814197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2819c25700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2814101dd0 0x7f2814197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2819c25700 1 -- 172.21.15.77:0/644385168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28141a6e80 con 0x7f28141064e0 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2818c23700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2814101dd0 0x7f2814197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f2819c25700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28141064e0 0x7f2814198000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f280c00b780 tx=0x7f280c004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f280a7fc700 1 -- 172.21.15.77:0/644385168 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c021070 con 0x7f28141064e0 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f280a7fc700 1 -- 172.21.15.77:0/644385168 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f280c007e30 con 0x7f28141064e0 2024-03-16T09:40:46.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28141a7110 con 0x7f28141064e0 2024-03-16T09:40:46.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.768+0000 7f280a7fc700 1 -- 172.21.15.77:0/644385168 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c004800 con 0x7f28141064e0 2024-03-16T09:40:46.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.769+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28141a7610 con 0x7f28141064e0 2024-03-16T09:40:46.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.770+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28141028e0 con 0x7f28141064e0 2024-03-16T09:40:46.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.771+0000 7f280a7fc700 1 -- 172.21.15.77:0/644385168 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f280c007980 con 0x7f28141064e0 2024-03-16T09:40:46.773 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:46.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.772+0000 7f280a7fc700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f28000a29a0 0x7f28000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:46.774 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:46.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.772+0000 7f2819424700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f28000a29a0 0x7f28000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:46.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.772+0000 7f280a7fc700 1 -- 172.21.15.77:0/644385168 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f280c00f020 con 0x7f28141064e0 2024-03-16T09:40:46.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.772+0000 7f2819424700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f28000a29a0 0x7f28000a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2810005950 tx=0x7f28100058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:46.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.776+0000 7f280a7fc700 1 -- 172.21.15.77:0/644385168 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f280c02a020 con 0x7f28141064e0 2024-03-16T09:40:46.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:46 smithi119 ceph-mon[35632]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T09:40:46.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:46 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1067420329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T09:40:46.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.979+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f281402db70 con 0x7f28141064e0 2024-03-16T09:40:46.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.979+0000 7f280a7fc700 1 -- 172.21.15.77:0/644385168 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f280c09d040 con 0x7f28141064e0 2024-03-16T09:40:46.982 INFO:teuthology.orchestra.run.smithi077.stdout:257698037798 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.981+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f28000a29a0 msgr2=0x7f28000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.981+0000 7f281b688700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f28000a29a0 0x7f28000a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2810005950 tx=0x7f28100058e0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28141064e0 msgr2=0x7f2814198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28141064e0 0x7f2814198000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f280c00b780 tx=0x7f280c004d00 comp rx=0 tx=0).stop 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 shutdown_connections 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f28000a29a0 0x7f28000a4e60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28141064e0 0x7f2814198000 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2814101420 0x7f2814197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 --2- 172.21.15.77:0/644385168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2814101dd0 0x7f2814197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 >> 172.21.15.77:0/644385168 conn(0x7f2814078140 msgr2=0x7f28140ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:46.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 shutdown_connections 2024-03-16T09:40:46.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:46.982+0000 7f281b688700 1 -- 172.21.15.77:0/644385168 wait complete. 2024-03-16T09:40:47.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:46 smithi129 ceph-mon[35841]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-16T09:40:47.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:46 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1067420329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T09:40:47.903 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:47 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/644385168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T09:40:47.940 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:47 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/644385168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T09:40:48.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:47 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/644385168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T09:40:48.229 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574917 got 55834574920 for osd.1 2024-03-16T09:40:48.229 DEBUG:teuthology.parallel:result is None 2024-03-16T09:40:48.649 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037794 got 257698037798 for osd.8 2024-03-16T09:40:48.650 DEBUG:teuthology.parallel:result is None 2024-03-16T09:40:48.890 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:48 smithi077 ceph-mon[32171]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:40:48.939 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:48 smithi119 ceph-mon[35632]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:40:49.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:48 smithi129 ceph-mon[35841]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:40:51.114 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:50 smithi077 ceph-mon[32171]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:51.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:50 smithi119 ceph-mon[35632]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:51.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:50 smithi129 ceph-mon[35841]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:51.891 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:52.496 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:40:52.917 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:52 smithi077 ceph-mon[32171]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:40:53.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:52 smithi119 ceph-mon[35632]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:40:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:52 smithi129 ceph-mon[35841]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:40:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1547574629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd40107660 msgr2=0x7fbd4010f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1547574629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd40107660 0x7fbd4010f1e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800b0a0 tx=0x7fbd3800b3b0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1547574629 shutdown_connections 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1547574629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd401025b0 0x7fbd40102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1547574629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd40107660 0x7fbd4010f1e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1547574629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd40102f60 0x7fbd40106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1547574629 >> 172.21.15.77:0/1547574629 conn(0x7fbd400793a0 msgr2=0x7fbd400797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.537+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1547574629 shutdown_connections 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1547574629 wait complete. 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 Processor -- start 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 -- start start 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd401025b0 0x7fbd4019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd40102f60 0x7fbd4019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40107660 0x7fbd40198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbd40112440 con 0x7fbd401025b0 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbd401122c0 con 0x7fbd40107660 2024-03-16T09:40:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd45d9d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd401125c0 con 0x7fbd40102f60 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd3f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd401025b0 0x7fbd4019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd3f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd401025b0 0x7fbd4019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38376/0 (socket says 172.21.15.77:38376) 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd3f7fe700 1 -- 172.21.15.77:0/1787402567 learned_addr learned my addr 172.21.15.77:0/1787402567 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd3ffff700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40107660 0x7fbd40198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.538+0000 7fbd37fff700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd40102f60 0x7fbd4019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3f7fe700 1 -- 172.21.15.77:0/1787402567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd40102f60 msgr2=0x7fbd4019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3f7fe700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd40102f60 0x7fbd4019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3f7fe700 1 -- 172.21.15.77:0/1787402567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40107660 msgr2=0x7fbd40198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3f7fe700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40107660 0x7fbd40198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3f7fe700 1 -- 172.21.15.77:0/1787402567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd40199000 con 0x7fbd401025b0 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3f7fe700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd401025b0 0x7fbd4019e600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000e9a0 tx=0x7fbd3000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3d7fa700 1 -- 172.21.15.77:0/1787402567 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000ca10 con 0x7fbd401025b0 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3d7fa700 1 -- 172.21.15.77:0/1787402567 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd3000cbb0 con 0x7fbd401025b0 2024-03-16T09:40:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd401992f0 con 0x7fbd401025b0 2024-03-16T09:40:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd3d7fa700 1 -- 172.21.15.77:0/1787402567 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30018680 con 0x7fbd401025b0 2024-03-16T09:40:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.539+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd401a5580 con 0x7fbd401025b0 2024-03-16T09:40:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.540+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd40103a10 con 0x7fbd401025b0 2024-03-16T09:40:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.541+0000 7fbd3d7fa700 1 -- 172.21.15.77:0/1787402567 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbd30010430 con 0x7fbd401025b0 2024-03-16T09:40:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.542+0000 7fbd3d7fa700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbd2c0a2920 0x7fbd2c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.542+0000 7fbd37fff700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbd2c0a2920 0x7fbd2c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.542+0000 7fbd3d7fa700 1 -- 172.21.15.77:0/1787402567 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fbd300cee10 con 0x7fbd401025b0 2024-03-16T09:40:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.542+0000 7fbd37fff700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbd2c0a2920 0x7fbd2c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbd40199e90 tx=0x7fbd28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:53.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.545+0000 7fbd3d7fa700 1 -- 172.21.15.77:0/1787402567 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbd3009bf00 con 0x7fbd401025b0 2024-03-16T09:40:53.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.745+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fbd4002db70 con 0x7fbd401025b0 2024-03-16T09:40:53.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.745+0000 7fbd3d7fa700 1 -- 172.21.15.77:0/1787402567 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbd3009b800 con 0x7fbd401025b0 2024-03-16T09:40:53.749 INFO:teuthology.orchestra.run.smithi077.stdout:77309411397 2024-03-16T09:40:53.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.747+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbd2c0a2920 msgr2=0x7fbd2c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:53.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.747+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbd2c0a2920 0x7fbd2c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbd40199e90 tx=0x7fbd28009260 comp rx=0 tx=0).stop 2024-03-16T09:40:53.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.747+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd401025b0 msgr2=0x7fbd4019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:53.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.747+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd401025b0 0x7fbd4019e600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000e9a0 tx=0x7fbd3000ecb0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 shutdown_connections 2024-03-16T09:40:53.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbd2c0a2920 0x7fbd2c0a4de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40107660 0x7fbd40198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd401025b0 0x7fbd4019e600 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 --2- 172.21.15.77:0/1787402567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd40102f60 0x7fbd4019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:53.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 >> 172.21.15.77:0/1787402567 conn(0x7fbd400793a0 msgr2=0x7fbd40100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:53.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 shutdown_connections 2024-03-16T09:40:53.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:53.748+0000 7fbd45d9d700 1 -- 172.21.15.77:0/1787402567 wait complete. 2024-03-16T09:40:54.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:53 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1787402567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T09:40:54.050 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:53 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1787402567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T09:40:54.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:53 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1787402567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T09:40:55.007 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:54 smithi077 ceph-mon[32171]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:55.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 -- 172.21.15.77:0/3120192978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7006a920 msgr2=0x7f8c7010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 --2- 172.21.15.77:0/3120192978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7006a920 0x7f8c7010f160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8c6800b0d0 tx=0x7f8c6800b3e0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 -- 172.21.15.77:0/3120192978 shutdown_connections 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 --2- 172.21.15.77:0/3120192978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7006a920 0x7f8c7010f160 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 --2- 172.21.15.77:0/3120192978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c70108320 0x7f8c70108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 --2- 172.21.15.77:0/3120192978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c70069f80 0x7f8c7006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 -- 172.21.15.77:0/3120192978 >> 172.21.15.77:0/3120192978 conn(0x7f8c70078100 msgr2=0x7f8c70078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 -- 172.21.15.77:0/3120192978 shutdown_connections 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.024+0000 7f8c78769700 1 -- 172.21.15.77:0/3120192978 wait complete. 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 Processor -- start 2024-03-16T09:40:55.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 -- start start 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70069f80 0x7f8c7019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c7006a920 0x7f8c7019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c70108320 0x7f8c70198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70112470 con 0x7f8c70069f80 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8c701122f0 con 0x7f8c7006a920 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c78769700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8c701125f0 con 0x7f8c70108320 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c76505700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70069f80 0x7f8c7019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c76505700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70069f80 0x7f8c7019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38402/0 (socket says 172.21.15.77:38402) 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c76505700 1 -- 172.21.15.77:0/1876676756 learned_addr learned my addr 172.21.15.77:0/1876676756 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c75d04700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c7006a920 0x7f8c7019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.025+0000 7f8c76d06700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c70108320 0x7f8c70198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c76505700 1 -- 172.21.15.77:0/1876676756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c70108320 msgr2=0x7f8c70198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c76505700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c70108320 0x7f8c70198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c76505700 1 -- 172.21.15.77:0/1876676756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c7006a920 msgr2=0x7f8c7019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c76505700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c7006a920 0x7f8c7019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c76505700 1 -- 172.21.15.77:0/1876676756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c70199030 con 0x7f8c70069f80 2024-03-16T09:40:55.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c76505700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70069f80 0x7f8c7019e680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c012db0 tx=0x7f8c6c00c5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:55.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c677fe700 1 -- 172.21.15.77:0/1876676756 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c01f070 con 0x7f8c70069f80 2024-03-16T09:40:55.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c677fe700 1 -- 172.21.15.77:0/1876676756 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c6c01a8e0 con 0x7f8c70069f80 2024-03-16T09:40:55.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c701992c0 con 0x7f8c70069f80 2024-03-16T09:40:55.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c677fe700 1 -- 172.21.15.77:0/1876676756 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c018800 con 0x7f8c70069f80 2024-03-16T09:40:55.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.026+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c701013a0 con 0x7f8c70069f80 2024-03-16T09:40:55.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.027+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c701039d0 con 0x7f8c70069f80 2024-03-16T09:40:55.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.028+0000 7f8c677fe700 1 -- 172.21.15.77:0/1876676756 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8c6c022070 con 0x7f8c70069f80 2024-03-16T09:40:55.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.029+0000 7f8c677fe700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8c5c0a2920 0x7f8c5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.029+0000 7f8c75d04700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8c5c0a2920 0x7f8c5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.030+0000 7f8c677fe700 1 -- 172.21.15.77:0/1876676756 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8c6c0d2bc0 con 0x7f8c70069f80 2024-03-16T09:40:55.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.030+0000 7f8c75d04700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8c5c0a2920 0x7f8c5c0a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8c70199f10 tx=0x7f8c60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:55.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.033+0000 7f8c677fe700 1 -- 172.21.15.77:0/1876676756 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c6c09fbb0 con 0x7f8c70069f80 2024-03-16T09:40:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:54 smithi119 ceph-mon[35632]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.240+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f8c70199c80 con 0x7f8c70069f80 2024-03-16T09:40:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.240+0000 7f8c677fe700 1 -- 172.21.15.77:0/1876676756 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8c6c09f4b0 con 0x7f8c70069f80 2024-03-16T09:40:55.243 INFO:teuthology.orchestra.run.smithi077.stdout:201863462960 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8c5c0a2920 msgr2=0x7f8c5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8c5c0a2920 0x7f8c5c0a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8c70199f10 tx=0x7f8c60009260 comp rx=0 tx=0).stop 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70069f80 msgr2=0x7f8c7019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70069f80 0x7f8c7019e680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c012db0 tx=0x7f8c6c00c5b0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 shutdown_connections 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8c5c0a2920 0x7f8c5c0a4de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c70108320 0x7f8c70198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c70069f80 0x7f8c7019e680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 --2- 172.21.15.77:0/1876676756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c7006a920 0x7f8c7019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 >> 172.21.15.77:0/1876676756 conn(0x7f8c70078100 msgr2=0x7f8c700ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 shutdown_connections 2024-03-16T09:40:55.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.243+0000 7f8c78769700 1 -- 172.21.15.77:0/1876676756 wait complete. 2024-03-16T09:40:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:54 smithi129 ceph-mon[35841]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:40:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 -- 172.21.15.77:0/160724842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac1011d0 msgr2=0x7f74ac101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 --2- 172.21.15.77:0/160724842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac1011d0 0x7f74ac101630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f749c009990 tx=0x7f749c009ca0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 -- 172.21.15.77:0/160724842 shutdown_connections 2024-03-16T09:40:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 --2- 172.21.15.77:0/160724842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74ac107210 0x7f74ac1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 --2- 172.21.15.77:0/160724842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74ac101b70 0x7f74ac10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 --2- 172.21.15.77:0/160724842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac1011d0 0x7f74ac101630 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 -- 172.21.15.77:0/160724842 >> 172.21.15.77:0/160724842 conn(0x7f74ac078100 msgr2=0x7f74ac078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 -- 172.21.15.77:0/160724842 shutdown_connections 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 -- 172.21.15.77:0/160724842 wait complete. 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.305+0000 7f74b3a48700 1 Processor -- start 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b3a48700 1 -- start start 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b3a48700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74ac1011d0 0x7f74ac199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b3a48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac101b70 0x7f74ac199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b3a48700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74ac107210 0x7f74ac1a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b3a48700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac111340 con 0x7f74ac101b70 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b3a48700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1111c0 con 0x7f74ac1011d0 2024-03-16T09:40:55.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b3a48700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1114c0 con 0x7f74ac107210 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac101b70 0x7f74ac199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac101b70 0x7f74ac199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38416/0 (socket says 172.21.15.77:38416) 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b17e4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74ac1011d0 0x7f74ac199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 -- 172.21.15.77:0/3036401632 learned_addr learned my addr 172.21.15.77:0/3036401632 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b1fe5700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74ac107210 0x7f74ac1a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 -- 172.21.15.77:0/3036401632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74ac107210 msgr2=0x7f74ac1a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74ac107210 0x7f74ac1a48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 -- 172.21.15.77:0/3036401632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74ac1011d0 msgr2=0x7f74ac199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74ac1011d0 0x7f74ac199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 -- 172.21.15.77:0/3036401632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ac1a4e30 con 0x7f74ac101b70 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b17e4700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74ac1011d0 0x7f74ac199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74b0fe3700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac101b70 0x7f74ac199db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f749c000c00 tx=0x7f749c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.306+0000 7f74a27fc700 1 -- 172.21.15.77:0/3036401632 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c027070 con 0x7f74ac101b70 2024-03-16T09:40:55.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.307+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ac1a5000 con 0x7f74ac101b70 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.307+0000 7f74a27fc700 1 -- 172.21.15.77:0/3036401632 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f749c022470 con 0x7f74ac101b70 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.307+0000 7f74a27fc700 1 -- 172.21.15.77:0/3036401632 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c00b610 con 0x7f74ac101b70 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.307+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ac1a54b0 con 0x7f74ac101b70 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.308+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74ac068f90 con 0x7f74ac101b70 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.309+0000 7f74a27fc700 1 -- 172.21.15.77:0/3036401632 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f749c02a070 con 0x7f74ac101b70 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.309+0000 7f74a27fc700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f74980a6d30 0x7f74980a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.310+0000 7f74b17e4700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f74980a6d30 0x7f74980a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.310+0000 7f74a27fc700 1 -- 172.21.15.77:0/3036401632 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f749c0cf6b0 con 0x7f74ac101b70 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.310+0000 7f74b17e4700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f74980a6d30 0x7f74980a91f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009fd0 tx=0x7f74a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.314+0000 7f74a27fc700 1 -- 172.21.15.77:0/3036401632 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f749c013070 con 0x7f74ac101b70 2024-03-16T09:40:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.520+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f74ac02db70 con 0x7f74ac101b70 2024-03-16T09:40:55.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.520+0000 7f74a27fc700 1 -- 172.21.15.77:0/3036401632 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f749c09c830 con 0x7f74ac101b70 2024-03-16T09:40:55.523 INFO:teuthology.orchestra.run.smithi077.stdout:313532612637 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f74980a6d30 msgr2=0x7f74980a91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f74980a6d30 0x7f74980a91f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009fd0 tx=0x7f74a8009260 comp rx=0 tx=0).stop 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac101b70 msgr2=0x7f74ac199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac101b70 0x7f74ac199db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f749c000c00 tx=0x7f749c004790 comp rx=0 tx=0).stop 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 shutdown_connections 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f74980a6d30 0x7f74980a91f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74ac1011d0 0x7f74ac199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74ac107210 0x7f74ac1a48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 --2- 172.21.15.77:0/3036401632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74ac101b70 0x7f74ac199db0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 >> 172.21.15.77:0/3036401632 conn(0x7f74ac078100 msgr2=0x7f74ac1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 shutdown_connections 2024-03-16T09:40:55.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:55.523+0000 7f74b3a48700 1 -- 172.21.15.77:0/3036401632 wait complete. 2024-03-16T09:40:55.939 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462955 got 201863462960 for osd.6 2024-03-16T09:40:55.939 DEBUG:teuthology.parallel:result is None 2024-03-16T09:40:56.083 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:55 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:40:56.083 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1876676756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T09:40:56.083 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3036401632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T09:40:56.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:55 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:40:56.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1876676756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T09:40:56.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3036401632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T09:40:56.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:55 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:40:56.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1876676756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T09:40:56.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3036401632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.677+0000 7f5b33628700 1 -- 172.21.15.77:0/898207987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 msgr2=0x7f5b2c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 --2- 172.21.15.77:0/898207987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c102800 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009990 tx=0x7f5b1c009ca0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 -- 172.21.15.77:0/898207987 shutdown_connections 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 --2- 172.21.15.77:0/898207987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b2c1083f0 0x7f5b2c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 --2- 172.21.15.77:0/898207987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b2c102d40 0x7f5b2c10f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 --2- 172.21.15.77:0/898207987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c102800 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 -- 172.21.15.77:0/898207987 >> 172.21.15.77:0/898207987 conn(0x7f5b2c0793a0 msgr2=0x7f5b2c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 -- 172.21.15.77:0/898207987 shutdown_connections 2024-03-16T09:40:56.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 -- 172.21.15.77:0/898207987 wait complete. 2024-03-16T09:40:56.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 Processor -- start 2024-03-16T09:40:56.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.678+0000 7f5b33628700 1 -- start start 2024-03-16T09:40:56.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b33628700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b33628700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b2c102d40 0x7f5b2c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b33628700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b2c1083f0 0x7f5b2c1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b33628700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c1124f0 con 0x7f5b2c102380 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b33628700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c112370 con 0x7f5b2c102d40 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b33628700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c112670 con 0x7f5b2c1083f0 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50936/0 (socket says 172.21.15.77:50936) 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 -- 172.21.15.77:0/1845190965 learned_addr learned my addr 172.21.15.77:0/1845190965 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b30bc3700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b2c102d40 0x7f5b2c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b31bc5700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b2c1083f0 0x7f5b2c1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 -- 172.21.15.77:0/1845190965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b2c1083f0 msgr2=0x7f5b2c1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b2c1083f0 0x7f5b2c1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 -- 172.21.15.77:0/1845190965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b2c102d40 msgr2=0x7f5b2c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b2c102d40 0x7f5b2c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 -- 172.21.15.77:0/1845190965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c198fb0 con 0x7f5b2c102380 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b313c4700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c19e650 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800e9a0 tx=0x7f5b2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b227fc700 1 -- 172.21.15.77:0/1845190965 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800ca10 con 0x7f5b2c102380 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b227fc700 1 -- 172.21.15.77:0/1845190965 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b2800cbb0 con 0x7f5b2c102380 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b227fc700 1 -- 172.21.15.77:0/1845190965 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b28018680 con 0x7f5b2c102380 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.679+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2c1992a0 con 0x7f5b2c102380 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.680+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2c1a54b0 con 0x7f5b2c102380 2024-03-16T09:40:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.681+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b2c10a7d0 con 0x7f5b2c102380 2024-03-16T09:40:56.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.682+0000 7f5b227fc700 1 -- 172.21.15.77:0/1845190965 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5b28010430 con 0x7f5b2c102380 2024-03-16T09:40:56.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.682+0000 7f5b227fc700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5b180a29a0 0x7f5b180a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:56.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.682+0000 7f5b30bc3700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5b180a29a0 0x7f5b180a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:56.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.682+0000 7f5b227fc700 1 -- 172.21.15.77:0/1845190965 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5b280ceb70 con 0x7f5b2c102380 2024-03-16T09:40:56.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.683+0000 7f5b30bc3700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5b180a29a0 0x7f5b180a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00b2e0 tx=0x7f5b1c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:56.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.686+0000 7f5b227fc700 1 -- 172.21.15.77:0/1845190965 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b2809bc60 con 0x7f5b2c102380 2024-03-16T09:40:56.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.889+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5b2c10a1b0 con 0x7f5b2c102380 2024-03-16T09:40:56.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.889+0000 7f5b227fc700 1 -- 172.21.15.77:0/1845190965 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5b2809b560 con 0x7f5b2c102380 2024-03-16T09:40:56.892 INFO:teuthology.orchestra.run.smithi077.stdout:167503724599 2024-03-16T09:40:56.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5b180a29a0 msgr2=0x7f5b180a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:56.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5b180a29a0 0x7f5b180a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00b2e0 tx=0x7f5b1c005960 comp rx=0 tx=0).stop 2024-03-16T09:40:56.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 msgr2=0x7f5b2c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:56.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c19e650 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800e9a0 tx=0x7f5b2800ecb0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 shutdown_connections 2024-03-16T09:40:56.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5b180a29a0 0x7f5b180a4e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b2c1083f0 0x7f5b2c1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b2c102380 0x7f5b2c19e650 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 --2- 172.21.15.77:0/1845190965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b2c102d40 0x7f5b2c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:56.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 >> 172.21.15.77:0/1845190965 conn(0x7f5b2c0793a0 msgr2=0x7f5b2c101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:56.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 shutdown_connections 2024-03-16T09:40:56.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:56.891+0000 7f5b33628700 1 -- 172.21.15.77:0/1845190965 wait complete. 2024-03-16T09:40:57.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612633 got 313532612637 for osd.10 2024-03-16T09:40:57.079 DEBUG:teuthology.parallel:result is None 2024-03-16T09:40:57.171 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:56 smithi077 ceph-mon[32171]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:40:57.171 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:56 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.171 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:56 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.171 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:56 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.172 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:56 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:56 smithi119 ceph-mon[35632]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:40:57.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:56 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:56 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:56 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:56 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:56 smithi129 ceph-mon[35841]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:40:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:56 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:56 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:56 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:56 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:40:57.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1845190965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T09:40:57.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-16T09:40:57.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-16T09:40:57.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-16T09:40:57.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T09:40:57.920 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-16T09:40:57.920 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-16T09:40:57.920 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T09:40:57.920 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:57 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1845190965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T09:40:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:57 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1845190965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T09:40:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:57 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-03-16T09:40:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:40:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-16T09:40:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-16T09:40:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-16T09:40:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T09:40:59.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-16T09:40:59.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-16T09:40:59.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T09:40:59.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-03-16T09:40:59.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T09:40:59.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/169537190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T09:40:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/169537190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T09:40:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T09:40:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/169537190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T09:40:59.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 -- 172.21.15.77:0/1419737633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4101b70 msgr2=0x7fb3a410e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:59.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 --2- 172.21.15.77:0/1419737633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4101b70 0x7fb3a410e050 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb394009990 tx=0x7fb394009ca0 comp rx=0 tx=0).stop 2024-03-16T09:40:59.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 -- 172.21.15.77:0/1419737633 shutdown_connections 2024-03-16T09:40:59.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 --2- 172.21.15.77:0/1419737633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3a4107210 0x7fb3a41075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:59.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 --2- 172.21.15.77:0/1419737633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4101b70 0x7fb3a410e050 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:59.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 --2- 172.21.15.77:0/1419737633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a41011d0 0x7fb3a4101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:59.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 -- 172.21.15.77:0/1419737633 >> 172.21.15.77:0/1419737633 conn(0x7fb3a4078100 msgr2=0x7fb3a4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:40:59.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 -- 172.21.15.77:0/1419737633 shutdown_connections 2024-03-16T09:40:59.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.972+0000 7fb3ac178700 1 -- 172.21.15.77:0/1419737633 wait complete. 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 Processor -- start 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 -- start start 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a41011d0 0x7fb3a4199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3a4101b70 0x7fb3a4199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4107210 0x7fb3a41a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a4111340 con 0x7fb3a4107210 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a41111c0 con 0x7fb3a4101b70 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3ac178700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a41114c0 con 0x7fb3a41011d0 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4107210 0x7fb3a41a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4107210 0x7fb3a41a48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50976/0 (socket says 172.21.15.77:50976) 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 -- 172.21.15.77:0/2799718023 learned_addr learned my addr 172.21.15.77:0/2799718023 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:40:59.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3a9713700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3a4101b70 0x7fb3a4199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3a9f14700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a41011d0 0x7fb3a4199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 -- 172.21.15.77:0/2799718023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a41011d0 msgr2=0x7fb3a4199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a41011d0 0x7fb3a4199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 -- 172.21.15.77:0/2799718023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3a4101b70 msgr2=0x7fb3a4199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3a4101b70 0x7fb3a4199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.973+0000 7fb3aa715700 1 -- 172.21.15.77:0/2799718023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a41a4de0 con 0x7fb3a4107210 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.974+0000 7fb3aa715700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4107210 0x7fb3a41a48a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb394004790 tx=0x7fb3940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.974+0000 7fb3a2ffd700 1 -- 172.21.15.77:0/2799718023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394027070 con 0x7fb3a4107210 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.974+0000 7fb3a2ffd700 1 -- 172.21.15.77:0/2799718023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb394022470 con 0x7fb3a4107210 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.974+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a41a4fb0 con 0x7fb3a4107210 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.974+0000 7fb3a2ffd700 1 -- 172.21.15.77:0/2799718023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39400b610 con 0x7fb3a4107210 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.974+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a41a5400 con 0x7fb3a4107210 2024-03-16T09:40:59.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.975+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a4068f90 con 0x7fb3a4107210 2024-03-16T09:40:59.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.976+0000 7fb3a2ffd700 1 -- 172.21.15.77:0/2799718023 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb39402a070 con 0x7fb3a4107210 2024-03-16T09:40:59.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.976+0000 7fb3a2ffd700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb38c0a2920 0x7fb38c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:40:59.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.977+0000 7fb3a9f14700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb38c0a2920 0x7fb38c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:40:59.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.977+0000 7fb3a2ffd700 1 -- 172.21.15.77:0/2799718023 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb3940cfcd0 con 0x7fb3a4107210 2024-03-16T09:40:59.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.977+0000 7fb3a9f14700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb38c0a2920 0x7fb38c0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00f540 tx=0x7fb39c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:40:59.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:59.980+0000 7fb3a2ffd700 1 -- 172.21.15.77:0/2799718023 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb394013070 con 0x7fb3a4107210 2024-03-16T09:41:00.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.563+0000 7f0716a73700 1 -- 172.21.15.77:0/651483577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710101690 msgr2=0x7f071010f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.563+0000 7f0716a73700 1 --2- 172.21.15.77:0/651483577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710101690 0x7f071010f220 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f070800b0d0 tx=0x7f070800b3e0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- 172.21.15.77:0/651483577 shutdown_connections 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 --2- 172.21.15.77:0/651483577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710101690 0x7f071010f220 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 --2- 172.21.15.77:0/651483577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07101073a0 0x7f0710107780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 --2- 172.21.15.77:0/651483577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0710100cd0 0x7f0710101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- 172.21.15.77:0/651483577 >> 172.21.15.77:0/651483577 conn(0x7f0710076520 msgr2=0x7f0710076930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- 172.21.15.77:0/651483577 shutdown_connections 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- 172.21.15.77:0/651483577 wait complete. 2024-03-16T09:41:00.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 Processor -- start 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- start start 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710100cd0 0x7f0710101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0710101690 0x7f07101023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07101073a0 0x7f0710102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f07101124f0 con 0x7f0710100cd0 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0710112370 con 0x7f0710101690 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f0716a73700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0710112670 con 0x7f07101073a0 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.564+0000 7f071480f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710100cd0 0x7f0710101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:00.091 INFO:teuthology.orchestra.run.smithi077.stdout:141733920829 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0710101690 0x7f07101023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f071480f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710100cd0 0x7f0710101eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50954/0 (socket says 172.21.15.77:50954) 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f0715010700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07101073a0 0x7f0710102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f071480f700 1 -- 172.21.15.77:0/169537190 learned_addr learned my addr 172.21.15.77:0/169537190 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070ffff700 1 -- 172.21.15.77:0/169537190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07101073a0 msgr2=0x7f0710102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070ffff700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07101073a0 0x7f0710102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070ffff700 1 -- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710100cd0 msgr2=0x7f0710101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070ffff700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710100cd0 0x7f0710101eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070ffff700 1 -- 172.21.15.77:0/169537190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07101a6ec0 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070ffff700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0710101690 0x7f07101023f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00e9f0 tx=0x7f06fc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070dffb700 1 -- 172.21.15.77:0/169537190 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc00c9c0 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07101a71b0 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070dffb700 1 -- 172.21.15.77:0/169537190 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06fc00cb60 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f070dffb700 1 -- 172.21.15.77:0/169537190 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc018680 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.565+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07101a7740 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.566+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f071010a8c0 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.568+0000 7f070dffb700 1 -- 172.21.15.77:0/169537190 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f06fc010430 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.568+0000 7f070dffb700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f07000a2920 0x7f07000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.568+0000 7f071480f700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f07000a2920 0x7f07000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.568+0000 7f070dffb700 1 -- 172.21.15.77:0/169537190 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f06fc0ceff0 con 0x7f0710101690 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.569+0000 7f071480f700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f07000a2920 0x7f07000a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0704007c10 tx=0x7f0704007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.576+0000 7f070dffb700 1 -- 172.21.15.77:0/169537190 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f06fc09bf40 con 0x7f0710101690 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.786+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0710103390 con 0x7f0710101690 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.787+0000 7f070dffb700 1 -- 172.21.15.77:0/169537190 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f06fc09b840 con 0x7f0710101690 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f07000a2920 msgr2=0x7f07000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f07000a2920 0x7f07000a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0704007c10 tx=0x7f0704007480 comp rx=0 tx=0).stop 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0710101690 msgr2=0x7f07101023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0710101690 0x7f07101023f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00e9f0 tx=0x7f06fc00ed00 comp rx=0 tx=0).stop 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 shutdown_connections 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f07000a2920 0x7f07000a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07101073a0 0x7f0710102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0710100cd0 0x7f0710101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 --2- 172.21.15.77:0/169537190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0710101690 0x7f07101023f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 >> 172.21.15.77:0/169537190 conn(0x7f0710076520 msgr2=0x7f07100fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:00.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 shutdown_connections 2024-03-16T09:41:00.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:40:58.789+0000 7f0716a73700 1 -- 172.21.15.77:0/169537190 wait complete. 2024-03-16T09:41:00.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.179+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb3a402db70 con 0x7fb3a4107210 2024-03-16T09:41:00.183 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:40:59 smithi077 ceph-mon[32171]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T09:41:00.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.180+0000 7fb3a2ffd700 1 -- 172.21.15.77:0/2799718023 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb39409cdd0 con 0x7fb3a4107210 2024-03-16T09:41:00.183 INFO:teuthology.orchestra.run.smithi077.stdout:223338299438 2024-03-16T09:41:00.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb38c0a2920 msgr2=0x7fb38c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:00.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb38c0a2920 0x7fb38c0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00f540 tx=0x7fb39c00b040 comp rx=0 tx=0).stop 2024-03-16T09:41:00.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4107210 msgr2=0x7fb3a41a48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:00.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4107210 0x7fb3a41a48a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb394004790 tx=0x7fb3940047c0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 shutdown_connections 2024-03-16T09:41:00.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb38c0a2920 0x7fb38c0a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3a4107210 0x7fb3a41a48a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a41011d0 0x7fb3a4199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 --2- 172.21.15.77:0/2799718023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3a4101b70 0x7fb3a4199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:00.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 >> 172.21.15.77:0/2799718023 conn(0x7fb3a4078100 msgr2=0x7fb3a40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:00.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 shutdown_connections 2024-03-16T09:41:00.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:00.182+0000 7fb3ac178700 1 -- 172.21.15.77:0/2799718023 wait complete. 2024-03-16T09:41:00.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:40:59 smithi119 ceph-mon[35632]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T09:41:00.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:40:59 smithi129 ceph-mon[35841]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T09:41:00.905 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920824 got 141733920829 for osd.4 2024-03-16T09:41:00.905 DEBUG:teuthology.parallel:result is None 2024-03-16T09:41:01.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 -- 172.21.15.77:0/4121188726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102ce0 msgr2=0x7f1e3010f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:01.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 --2- 172.21.15.77:0/4121188726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102ce0 0x7f1e3010f120 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800b0d0 tx=0x7f1e2800b3e0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 -- 172.21.15.77:0/4121188726 shutdown_connections 2024-03-16T09:41:01.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 --2- 172.21.15.77:0/4121188726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e30108300 0x7f1e301086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 --2- 172.21.15.77:0/4121188726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102ce0 0x7f1e3010f120 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 --2- 172.21.15.77:0/4121188726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e30102320 0x7f1e301027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 -- 172.21.15.77:0/4121188726 >> 172.21.15.77:0/4121188726 conn(0x7f1e30079400 msgr2=0x7f1e30079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 -- 172.21.15.77:0/4121188726 shutdown_connections 2024-03-16T09:41:01.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.103+0000 7f1e370d4700 1 -- 172.21.15.77:0/4121188726 wait complete. 2024-03-16T09:41:01.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 Processor -- start 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 -- start start 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102320 0x7f1e3019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e30102ce0 0x7f1e3019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e30108300 0x7f1e30198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1e30112430 con 0x7f1e30102320 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1e301122b0 con 0x7f1e30102ce0 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e370d4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1e301125b0 con 0x7f1e30108300 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e34e70700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102320 0x7f1e3019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e34e70700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102320 0x7f1e3019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50994/0 (socket says 172.21.15.77:50994) 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e34e70700 1 -- 172.21.15.77:0/2294596253 learned_addr learned my addr 172.21.15.77:0/2294596253 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e2ffff700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e30102ce0 0x7f1e3019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.104+0000 7f1e35671700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e30108300 0x7f1e30198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e34e70700 1 -- 172.21.15.77:0/2294596253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e30108300 msgr2=0x7f1e30198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:01.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e34e70700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e30108300 0x7f1e30198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e34e70700 1 -- 172.21.15.77:0/2294596253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e30102ce0 msgr2=0x7f1e3019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e34e70700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e30102ce0 0x7f1e3019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e34e70700 1 -- 172.21.15.77:0/2294596253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e30199030 con 0x7f1e30102320 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e34e70700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102320 0x7f1e3019e680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400e9a0 tx=0x7f1e2400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e2dffb700 1 -- 172.21.15.77:0/2294596253 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2400ca10 con 0x7f1e30102320 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e2dffb700 1 -- 172.21.15.77:0/2294596253 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e2400cbb0 con 0x7f1e30102320 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e30199320 con 0x7f1e30102320 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e2dffb700 1 -- 172.21.15.77:0/2294596253 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e24018680 con 0x7f1e30102320 2024-03-16T09:41:01.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.105+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e301a5600 con 0x7f1e30102320 2024-03-16T09:41:01.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.106+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e301039b0 con 0x7f1e30102320 2024-03-16T09:41:01.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.107+0000 7f1e2dffb700 1 -- 172.21.15.77:0/2294596253 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1e24010430 con 0x7f1e30102320 2024-03-16T09:41:01.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.108+0000 7f1e2dffb700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1e200a29a0 0x7f1e200a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:01.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.108+0000 7f1e2ffff700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1e200a29a0 0x7f1e200a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:01.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.108+0000 7f1e2dffb700 1 -- 172.21.15.77:0/2294596253 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1e240ceca0 con 0x7f1e30102320 2024-03-16T09:41:01.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.108+0000 7f1e2ffff700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1e200a29a0 0x7f1e200a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1e30199f10 tx=0x7f1e1c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.112+0000 7f1e2dffb700 1 -- 172.21.15.77:0/2294596253 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e2409bbf0 con 0x7f1e30102320 2024-03-16T09:41:01.127 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:00 smithi077 ceph-mon[32171]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T09:41:01.127 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:00 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2799718023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T09:41:01.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:00 smithi119 ceph-mon[35632]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T09:41:01.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:00 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2799718023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T09:41:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:00 smithi129 ceph-mon[35841]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T09:41:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:00 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2799718023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T09:41:01.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.313+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1e30199c80 con 0x7f1e30102320 2024-03-16T09:41:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.313+0000 7f1e2dffb700 1 -- 172.21.15.77:0/2294596253 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1e2409b4f0 con 0x7f1e30102320 2024-03-16T09:41:01.316 INFO:teuthology.orchestra.run.smithi077.stdout:34359738448 2024-03-16T09:41:01.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1e200a29a0 msgr2=0x7f1e200a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:01.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1e200a29a0 0x7f1e200a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1e30199f10 tx=0x7f1e1c0073d0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102320 msgr2=0x7f1e3019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:01.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102320 0x7f1e3019e680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400e9a0 tx=0x7f1e2400ecb0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 shutdown_connections 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1e200a29a0 0x7f1e200a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e30108300 0x7f1e30198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e30102320 0x7f1e3019e680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 --2- 172.21.15.77:0/2294596253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e30102ce0 0x7f1e3019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 >> 172.21.15.77:0/2294596253 conn(0x7f1e30079400 msgr2=0x7f1e30101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 shutdown_connections 2024-03-16T09:41:01.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:01.315+0000 7f1e370d4700 1 -- 172.21.15.77:0/2294596253 wait complete. 2024-03-16T09:41:01.952 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:01 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2294596253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T09:41:01.982 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411391 got 77309411397 for osd.2 2024-03-16T09:41:01.982 DEBUG:teuthology.parallel:result is None 2024-03-16T09:41:02.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:01 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2294596253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T09:41:02.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:01 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2294596253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T09:41:02.790 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724594 got 167503724599 for osd.5 2024-03-16T09:41:02.790 DEBUG:teuthology.parallel:result is None 2024-03-16T09:41:03.176 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:02 smithi077 ceph-mon[32171]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T09:41:03.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:02 smithi119 ceph-mon[35632]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T09:41:03.211 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299432 got 223338299438 for osd.7 2024-03-16T09:41:03.211 DEBUG:teuthology.parallel:result is None 2024-03-16T09:41:03.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:02 smithi129 ceph-mon[35841]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 -- 172.21.15.77:0/686297322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 msgr2=0x7f97cc101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 --2- 172.21.15.77:0/686297322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc101630 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009990 tx=0x7f97bc009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 -- 172.21.15.77:0/686297322 shutdown_connections 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 --2- 172.21.15.77:0/686297322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97cc101b70 0x7f97cc10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 --2- 172.21.15.77:0/686297322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc107210 0x7f97cc1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 --2- 172.21.15.77:0/686297322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc101630 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 -- 172.21.15.77:0/686297322 >> 172.21.15.77:0/686297322 conn(0x7f97cc078100 msgr2=0x7f97cc078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 -- 172.21.15.77:0/686297322 shutdown_connections 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 -- 172.21.15.77:0/686297322 wait complete. 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.335+0000 7f97d3d36700 1 Processor -- start 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 -- start start 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:04.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97cc101b70 0x7f97cc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc107210 0x7f97cc1a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc111320 con 0x7f97cc1011d0 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1111a0 con 0x7f97cc101b70 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1114a0 con 0x7f97cc107210 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51020/0 (socket says 172.21.15.77:51020) 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 -- 172.21.15.77:0/667761258 learned_addr learned my addr 172.21.15.77:0/667761258 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d12d1700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97cc101b70 0x7f97cc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d22d3700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc107210 0x7f97cc1a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 -- 172.21.15.77:0/667761258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc107210 msgr2=0x7f97cc1a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc107210 0x7f97cc1a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 -- 172.21.15.77:0/667761258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97cc101b70 msgr2=0x7f97cc199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97cc101b70 0x7f97cc199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 -- 172.21.15.77:0/667761258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc1a4dc0 con 0x7f97cc1011d0 2024-03-16T09:41:04.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d1ad2700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc199800 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f97c800d950 tx=0x7f97c800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97c2ffd700 1 -- 172.21.15.77:0/667761258 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8016070 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97c2ffd700 1 -- 172.21.15.77:0/667761258 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97c800fd40 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97cc1a4ff0 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.336+0000 7f97c2ffd700 1 -- 172.21.15.77:0/667761258 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c800e3b0 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.337+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97cc1a53f0 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.337+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97cc068f90 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.338+0000 7f97c2ffd700 1 -- 172.21.15.77:0/667761258 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f97c8019070 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.339+0000 7f97c2ffd700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f97b80a2920 0x7f97b80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.339+0000 7f97d12d1700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f97b80a2920 0x7f97b80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.339+0000 7f97c2ffd700 1 -- 172.21.15.77:0/667761258 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f97c80ce580 con 0x7f97cc1011d0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.339+0000 7f97d12d1700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f97b80a2920 0x7f97b80a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00b5c0 tx=0x7f97bc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:04.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.343+0000 7f97c2ffd700 1 -- 172.21.15.77:0/667761258 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97c809b4d0 con 0x7f97cc1011d0 2024-03-16T09:41:04.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.538+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f97cc02dc90 con 0x7f97cc1011d0 2024-03-16T09:41:04.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.539+0000 7f97c2ffd700 1 -- 172.21.15.77:0/667761258 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f97c809add0 con 0x7f97cc1011d0 2024-03-16T09:41:04.542 INFO:teuthology.orchestra.run.smithi077.stdout:292057776165 2024-03-16T09:41:04.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f97b80a2920 msgr2=0x7f97b80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:04.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f97b80a2920 0x7f97b80a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00b5c0 tx=0x7f97bc005960 comp rx=0 tx=0).stop 2024-03-16T09:41:04.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 msgr2=0x7f97cc199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:04.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc199800 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f97c800d950 tx=0x7f97c800dc60 comp rx=0 tx=0).stop 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 shutdown_connections 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f97b80a2920 0x7f97b80a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc107210 0x7f97cc1a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97cc1011d0 0x7f97cc199800 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 --2- 172.21.15.77:0/667761258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97cc101b70 0x7f97cc199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 >> 172.21.15.77:0/667761258 conn(0x7f97cc078100 msgr2=0x7f97cc0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 shutdown_connections 2024-03-16T09:41:04.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:04.541+0000 7f97d3d36700 1 -- 172.21.15.77:0/667761258 wait complete. 2024-03-16T09:41:04.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738448 for osd.0 2024-03-16T09:41:04.963 DEBUG:teuthology.parallel:result is None 2024-03-16T09:41:05.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 -- 172.21.15.77:0/3457356736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194101e10 msgr2=0x7f7194105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:05.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 --2- 172.21.15.77:0/3457356736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194101e10 0x7f7194105e10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7180009990 tx=0x7f7180009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:04 smithi077 ceph-mon[32171]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:05.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:04 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/667761258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T09:41:05.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 -- 172.21.15.77:0/3457356736 shutdown_connections 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 --2- 172.21.15.77:0/3457356736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71941064f0 0x7f719410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 --2- 172.21.15.77:0/3457356736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 0x7f7194101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 --2- 172.21.15.77:0/3457356736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7194101e10 0x7f7194105e10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 -- 172.21.15.77:0/3457356736 >> 172.21.15.77:0/3457356736 conn(0x7f71940771f0 msgr2=0x7f7194077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 -- 172.21.15.77:0/3457356736 shutdown_connections 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.144+0000 7f719ad37700 1 -- 172.21.15.77:0/3457356736 wait complete. 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 Processor -- start 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 -- start start 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 0x7f71941a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7194101e10 0x7f71941a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71941064f0 0x7f719419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7194111340 con 0x7f71941064f0 2024-03-16T09:41:05.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f71941111c0 con 0x7f7194101e10 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f719ad37700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71941114c0 con 0x7f7194101460 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f7198ad3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 0x7f71941a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f7193fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7194101e10 0x7f71941a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f7198ad3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 0x7f71941a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:43654/0 (socket says 172.21.15.77:43654) 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f7198ad3700 1 -- 172.21.15.77:0/4285445149 learned_addr learned my addr 172.21.15.77:0/4285445149 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.145+0000 7f71992d4700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71941064f0 0x7f719419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7198ad3700 1 -- 172.21.15.77:0/4285445149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7194101e10 msgr2=0x7f71941a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7198ad3700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7194101e10 0x7f71941a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7198ad3700 1 -- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71941064f0 msgr2=0x7f719419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7198ad3700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71941064f0 0x7f719419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7198ad3700 1 -- 172.21.15.77:0/4285445149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719419b140 con 0x7f7194101460 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7193fff700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7194101e10 0x7f71941a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f71992d4700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71941064f0 0x7f719419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7198ad3700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 0x7f71941a0820 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f718800d980 tx=0x7f718800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:05.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7191ffb700 1 -- 172.21.15.77:0/4285445149 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188016070 con 0x7f7194101460 2024-03-16T09:41:05.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7191ffb700 1 -- 172.21.15.77:0/4285445149 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7188004830 con 0x7f7194101460 2024-03-16T09:41:05.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719419b430 con 0x7f7194101460 2024-03-16T09:41:05.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f7191ffb700 1 -- 172.21.15.77:0/4285445149 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188005530 con 0x7f7194101460 2024-03-16T09:41:05.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.146+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71941a76f0 con 0x7f7194101460 2024-03-16T09:41:05.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.147+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7194068f90 con 0x7f7194101460 2024-03-16T09:41:05.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.148+0000 7f7191ffb700 1 -- 172.21.15.77:0/4285445149 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f71880049e0 con 0x7f7194101460 2024-03-16T09:41:05.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.149+0000 7f7191ffb700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71840a2920 0x7f71840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:05.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.149+0000 7f7193fff700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71840a2920 0x7f71840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:05.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.149+0000 7f7191ffb700 1 -- 172.21.15.77:0/4285445149 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f71880ce960 con 0x7f7194101460 2024-03-16T09:41:05.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.149+0000 7f7193fff700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71840a2920 0x7f71840a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f718000b2e0 tx=0x7f7180005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:05.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.153+0000 7f7191ffb700 1 -- 172.21.15.77:0/4285445149 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f718801f020 con 0x7f7194101460 2024-03-16T09:41:05.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:04 smithi119 ceph-mon[35632]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:05.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:04 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/667761258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T09:41:05.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:04 smithi129 ceph-mon[35841]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:05.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:04 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/667761258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T09:41:05.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.352+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f719402db70 con 0x7f7194101460 2024-03-16T09:41:05.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.352+0000 7f7191ffb700 1 -- 172.21.15.77:0/4285445149 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7188022350 con 0x7f7194101460 2024-03-16T09:41:05.355 INFO:teuthology.orchestra.run.smithi077.stdout:107374182466 2024-03-16T09:41:05.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71840a2920 msgr2=0x7f71840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:05.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71840a2920 0x7f71840a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f718000b2e0 tx=0x7f7180005960 comp rx=0 tx=0).stop 2024-03-16T09:41:05.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 msgr2=0x7f71941a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:05.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 0x7f71941a0820 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f718800d980 tx=0x7f718800dc90 comp rx=0 tx=0).stop 2024-03-16T09:41:05.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 shutdown_connections 2024-03-16T09:41:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71840a2920 0x7f71840a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71941064f0 0x7f719419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7194101460 0x7f71941a0820 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 --2- 172.21.15.77:0/4285445149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7194101e10 0x7f71941a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 >> 172.21.15.77:0/4285445149 conn(0x7f71940771f0 msgr2=0x7f7194107670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 shutdown_connections 2024-03-16T09:41:05.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:05.354+0000 7f719ad37700 1 -- 172.21.15.77:0/4285445149 wait complete. 2024-03-16T09:41:06.052 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776159 got 292057776165 for osd.9 2024-03-16T09:41:06.052 DEBUG:teuthology.parallel:result is None 2024-03-16T09:41:06.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:05 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4285445149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T09:41:06.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:05 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4285445149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T09:41:06.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:05 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4285445149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T09:41:06.400 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182461 got 107374182466 for osd.3 2024-03-16T09:41:06.401 DEBUG:teuthology.parallel:result is None 2024-03-16T09:41:06.401 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-16T09:41:06.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T09:41:07.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:06 smithi077 ceph-mon[32171]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:07.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:06 smithi119 ceph-mon[35632]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:07.215 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:07.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:06 smithi129 ceph-mon[35841]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:08.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.889+0000 7f0edf106700 1 -- 172.21.15.77:0/2264231321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed81072c0 msgr2=0x7f0ed806a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.889+0000 7f0edf106700 1 --2- 172.21.15.77:0/2264231321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed81072c0 0x7f0ed806a9f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009990 tx=0x7f0ecc009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- 172.21.15.77:0/2264231321 shutdown_connections 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 --2- 172.21.15.77:0/2264231321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed8104cb0 0x7f0ed8105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 --2- 172.21.15.77:0/2264231321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed81072c0 0x7f0ed806a9f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 --2- 172.21.15.77:0/2264231321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ed806af30 0x7f0ed806b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- 172.21.15.77:0/2264231321 >> 172.21.15.77:0/2264231321 conn(0x7f0ed80781e0 msgr2=0x7f0ed80785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- 172.21.15.77:0/2264231321 shutdown_connections 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- 172.21.15.77:0/2264231321 wait complete. 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 Processor -- start 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- start start 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed806af30 0x7f0ed819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed8104cb0 0x7f0ed819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ed81072c0 0x7f0ed81a3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:08.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8118970 con 0x7f0ed806af30 2024-03-16T09:41:08.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81187f0 con 0x7f0ed8104cb0 2024-03-16T09:41:08.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.890+0000 7f0edf106700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8118af0 con 0x7f0ed81072c0 2024-03-16T09:41:08.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed806af30 0x7f0ed819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:08.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed806af30 0x7f0ed819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45592/0 (socket says 172.21.15.77:45592) 2024-03-16T09:41:08.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 -- 172.21.15.77:0/1109459119 learned_addr learned my addr 172.21.15.77:0/1109459119 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:08.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0ed7fff700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed8104cb0 0x7f0ed819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:08.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edd6a3700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ed81072c0 0x7f0ed81a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 -- 172.21.15.77:0/1109459119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ed81072c0 msgr2=0x7f0ed81a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ed81072c0 0x7f0ed81a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 -- 172.21.15.77:0/1109459119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed8104cb0 msgr2=0x7f0ed819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed8104cb0 0x7f0ed819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 -- 172.21.15.77:0/1109459119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed81a3f50 con 0x7f0ed806af30 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.891+0000 7f0edcea2700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed806af30 0x7f0ed819ef40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000c00 tx=0x7f0ecc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.893+0000 7f0ed5ffb700 1 -- 172.21.15.77:0/1109459119 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc027070 con 0x7f0ed806af30 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.893+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed81a41e0 con 0x7f0ed806af30 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.893+0000 7f0ed5ffb700 1 -- 172.21.15.77:0/1109459119 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ecc022470 con 0x7f0ed806af30 2024-03-16T09:41:08.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.893+0000 7f0ed5ffb700 1 -- 172.21.15.77:0/1109459119 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc017650 con 0x7f0ed806af30 2024-03-16T09:41:08.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.893+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed81abac0 con 0x7f0ed806af30 2024-03-16T09:41:08.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.894+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed8068f90 con 0x7f0ed806af30 2024-03-16T09:41:08.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.895+0000 7f0ed5ffb700 1 -- 172.21.15.77:0/1109459119 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0ecc02a070 con 0x7f0ed806af30 2024-03-16T09:41:08.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.896+0000 7f0ed5ffb700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ec80a2920 0x7f0ec80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:08.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.896+0000 7f0ed5ffb700 1 -- 172.21.15.77:0/1109459119 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0ecc0cfae0 con 0x7f0ed806af30 2024-03-16T09:41:08.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.896+0000 7f0ed7fff700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ec80a2920 0x7f0ec80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:08.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.896+0000 7f0ed7fff700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ec80a2920 0x7f0ec80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8078940 tx=0x7f0ec40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:08.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:08.900+0000 7f0ed5ffb700 1 -- 172.21.15.77:0/1109459119 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ecc013070 con 0x7f0ed806af30 2024-03-16T09:41:09.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:08 smithi077 ceph-mon[32171]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:09.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.124+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ed8103220 con 0x7f0ec80a2920 2024-03-16T09:41:09.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.139+0000 7f0ed5ffb700 1 -- 172.21.15.77:0/1109459119 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258687 (secure 0 0 0) 0x7f0ed8103220 con 0x7f0ec80a2920 2024-03-16T09:41:09.146 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ec80a2920 msgr2=0x7f0ec80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ec80a2920 0x7f0ec80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8078940 tx=0x7f0ec40073d0 comp rx=0 tx=0).stop 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed806af30 msgr2=0x7f0ed819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed806af30 0x7f0ed819ef40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000c00 tx=0x7f0ecc0047b0 comp rx=0 tx=0).stop 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 shutdown_connections 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ec80a2920 0x7f0ec80a4de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ed81072c0 0x7f0ed81a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:09.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ed806af30 0x7f0ed819ef40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 --2- 172.21.15.77:0/1109459119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed8104cb0 0x7f0ed819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 >> 172.21.15.77:0/1109459119 conn(0x7f0ed80781e0 msgr2=0x7f0ed8102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 shutdown_connections 2024-03-16T09:41:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:09.143+0000 7f0edf106700 1 -- 172.21.15.77:0/1109459119 wait complete. 2024-03-16T09:41:09.152 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T09:41:09.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:08 smithi119 ceph-mon[35632]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:09.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:08 smithi129 ceph-mon[35841]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:09.740 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-03-16T09:41:07.788747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654235,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":549,"num_read_kb":387,"num_write":338,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":294,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358380,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583444,"statfs":{"total":1151940427776,"available":1151573446656,"internally_reserved":0,"allocated":13627392,"data_stored":7484013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.930633"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.861250+0000","last_change":"2024-03-16T09:39:56.184389+0000","last_active":"2024-03-16T09:40:58.861250+0000","last_peered":"2024-03-16T09:40:58.861250+0000","last_clean":"2024-03-16T09:40:58.861250+0000","last_became_active":"2024-03-16T09:39:56.184129+0000","last_became_peered":"2024-03-16T09:39:56.184129+0000","last_unstale":"2024-03-16T09:40:58.861250+0000","last_undegraded":"2024-03-16T09:40:58.861250+0000","last_fullsized":"2024-03-16T09:40:58.861250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:33.786788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865568+0000","last_change":"2024-03-16T09:39:56.193451+0000","last_active":"2024-03-16T09:40:58.865568+0000","last_peered":"2024-03-16T09:40:58.865568+0000","last_clean":"2024-03-16T09:40:58.865568+0000","last_became_active":"2024-03-16T09:39:56.187381+0000","last_became_peered":"2024-03-16T09:39:56.187381+0000","last_unstale":"2024-03-16T09:40:58.865568+0000","last_undegraded":"2024-03-16T09:40:58.865568+0000","last_fullsized":"2024-03-16T09:40:58.865568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:51.915654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865317+0000","last_change":"2024-03-16T09:39:56.183785+0000","last_active":"2024-03-16T09:40:58.865317+0000","last_peered":"2024-03-16T09:40:58.865317+0000","last_clean":"2024-03-16T09:40:58.865317+0000","last_became_active":"2024-03-16T09:39:56.183421+0000","last_became_peered":"2024-03-16T09:39:56.183421+0000","last_unstale":"2024-03-16T09:40:58.865317+0000","last_undegraded":"2024-03-16T09:40:58.865317+0000","last_fullsized":"2024-03-16T09:40:58.865317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:47.259139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863555+0000","last_change":"2024-03-16T09:39:56.195743+0000","last_active":"2024-03-16T09:40:58.863555+0000","last_peered":"2024-03-16T09:40:58.863555+0000","last_clean":"2024-03-16T09:40:58.863555+0000","last_became_active":"2024-03-16T09:39:56.194818+0000","last_became_peered":"2024-03-16T09:39:56.194818+0000","last_unstale":"2024-03-16T09:40:58.863555+0000","last_undegraded":"2024-03-16T09:40:58.863555+0000","last_fullsized":"2024-03-16T09:40:58.863555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:39:43.123879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863606+0000","last_change":"2024-03-16T09:39:56.184970+0000","last_active":"2024-03-16T09:40:58.863606+0000","last_peered":"2024-03-16T09:40:58.863606+0000","last_clean":"2024-03-16T09:40:58.863606+0000","last_became_active":"2024-03-16T09:39:56.184632+0000","last_became_peered":"2024-03-16T09:39:56.184632+0000","last_unstale":"2024-03-16T09:40:58.863606+0000","last_undegraded":"2024-03-16T09:40:58.863606+0000","last_fullsized":"2024-03-16T09:40:58.863606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:46:51.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.218928+0000","last_change":"2024-03-16T09:39:56.199688+0000","last_active":"2024-03-16T09:40:59.218928+0000","last_peered":"2024-03-16T09:40:59.218928+0000","last_clean":"2024-03-16T09:40:59.218928+0000","last_became_active":"2024-03-16T09:39:56.199094+0000","last_became_peered":"2024-03-16T09:39:56.199094+0000","last_unstale":"2024-03-16T09:40:59.218928+0000","last_undegraded":"2024-03-16T09:40:59.218928+0000","last_fullsized":"2024-03-16T09:40:59.218928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:10.855504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860444+0000","last_change":"2024-03-16T09:39:56.180816+0000","last_active":"2024-03-16T09:40:58.860444+0000","last_peered":"2024-03-16T09:40:58.860444+0000","last_clean":"2024-03-16T09:40:58.860444+0000","last_became_active":"2024-03-16T09:39:56.180499+0000","last_became_peered":"2024-03-16T09:39:56.180499+0000","last_unstale":"2024-03-16T09:40:58.860444+0000","last_undegraded":"2024-03-16T09:40:58.860444+0000","last_fullsized":"2024-03-16T09:40:58.860444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:32:09.444494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871439+0000","last_change":"2024-03-16T09:39:56.189960+0000","last_active":"2024-03-16T09:40:58.871439+0000","last_peered":"2024-03-16T09:40:58.871439+0000","last_clean":"2024-03-16T09:40:58.871439+0000","last_became_active":"2024-03-16T09:39:56.189609+0000","last_became_peered":"2024-03-16T09:39:56.189609+0000","last_unstale":"2024-03-16T09:40:58.871439+0000","last_undegraded":"2024-03-16T09:40:58.871439+0000","last_fullsized":"2024-03-16T09:40:58.871439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:50.219703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.867996+0000","last_change":"2024-03-16T09:40:58.867996+0000","last_active":"2024-03-16T09:40:58.867996+0000","last_peered":"2024-03-16T09:40:58.867996+0000","last_clean":"2024-03-16T09:40:58.867996+0000","last_became_active":"2024-03-16T09:40:58.867619+0000","last_became_peered":"2024-03-16T09:40:58.867619+0000","last_unstale":"2024-03-16T09:40:58.867996+0000","last_undegraded":"2024-03-16T09:40:58.867996+0000","last_fullsized":"2024-03-16T09:40:58.867996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:36.972983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.219160+0000","last_change":"2024-03-16T09:39:56.201584+0000","last_active":"2024-03-16T09:40:59.219160+0000","last_peered":"2024-03-16T09:40:59.219160+0000","last_clean":"2024-03-16T09:40:59.219160+0000","last_became_active":"2024-03-16T09:39:56.201320+0000","last_became_peered":"2024-03-16T09:39:56.201320+0000","last_unstale":"2024-03-16T09:40:59.219160+0000","last_undegraded":"2024-03-16T09:40:59.219160+0000","last_fullsized":"2024-03-16T09:40:59.219160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:18:00.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860869+0000","last_change":"2024-03-16T09:39:56.191125+0000","last_active":"2024-03-16T09:40:58.860869+0000","last_peered":"2024-03-16T09:40:58.860869+0000","last_clean":"2024-03-16T09:40:58.860869+0000","last_became_active":"2024-03-16T09:39:56.190795+0000","last_became_peered":"2024-03-16T09:39:56.190795+0000","last_unstale":"2024-03-16T09:40:58.860869+0000","last_undegraded":"2024-03-16T09:40:58.860869+0000","last_fullsized":"2024-03-16T09:40:58.860869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:44:10.894995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863425+0000","last_change":"2024-03-16T09:39:56.195652+0000","last_active":"2024-03-16T09:40:58.863425+0000","last_peered":"2024-03-16T09:40:58.863425+0000","last_clean":"2024-03-16T09:40:58.863425+0000","last_became_active":"2024-03-16T09:39:56.195270+0000","last_became_peered":"2024-03-16T09:39:56.195270+0000","last_unstale":"2024-03-16T09:40:58.863425+0000","last_undegraded":"2024-03-16T09:40:58.863425+0000","last_fullsized":"2024-03-16T09:40:58.863425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:03:41.546991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866140+0000","last_change":"2024-03-16T09:39:56.184622+0000","last_active":"2024-03-16T09:40:58.866140+0000","last_peered":"2024-03-16T09:40:58.866140+0000","last_clean":"2024-03-16T09:40:58.866140+0000","last_became_active":"2024-03-16T09:39:56.183532+0000","last_became_peered":"2024-03-16T09:39:56.183532+0000","last_unstale":"2024-03-16T09:40:58.866140+0000","last_undegraded":"2024-03-16T09:40:58.866140+0000","last_fullsized":"2024-03-16T09:40:58.866140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:11:24.574088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865139+0000","last_change":"2024-03-16T09:39:56.184393+0000","last_active":"2024-03-16T09:40:58.865139+0000","last_peered":"2024-03-16T09:40:58.865139+0000","last_clean":"2024-03-16T09:40:58.865139+0000","last_became_active":"2024-03-16T09:39:56.183783+0000","last_became_peered":"2024-03-16T09:39:56.183783+0000","last_unstale":"2024-03-16T09:40:58.865139+0000","last_undegraded":"2024-03-16T09:40:58.865139+0000","last_fullsized":"2024-03-16T09:40:58.865139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:16:22.131116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864979+0000","last_change":"2024-03-16T09:39:56.197365+0000","last_active":"2024-03-16T09:40:58.864979+0000","last_peered":"2024-03-16T09:40:58.864979+0000","last_clean":"2024-03-16T09:40:58.864979+0000","last_became_active":"2024-03-16T09:39:56.197020+0000","last_became_peered":"2024-03-16T09:39:56.197020+0000","last_unstale":"2024-03-16T09:40:58.864979+0000","last_undegraded":"2024-03-16T09:40:58.864979+0000","last_fullsized":"2024-03-16T09:40:58.864979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:29.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.862940+0000","last_change":"2024-03-16T09:39:56.185513+0000","last_active":"2024-03-16T09:40:58.862940+0000","last_peered":"2024-03-16T09:40:58.862940+0000","last_clean":"2024-03-16T09:40:58.862940+0000","last_became_active":"2024-03-16T09:39:56.184384+0000","last_became_peered":"2024-03-16T09:39:56.184384+0000","last_unstale":"2024-03-16T09:40:58.862940+0000","last_undegraded":"2024-03-16T09:40:58.862940+0000","last_fullsized":"2024-03-16T09:40:58.862940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:29.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.861161+0000","last_change":"2024-03-16T09:39:56.184536+0000","last_active":"2024-03-16T09:40:58.861161+0000","last_peered":"2024-03-16T09:40:58.861161+0000","last_clean":"2024-03-16T09:40:58.861161+0000","last_became_active":"2024-03-16T09:39:56.184208+0000","last_became_peered":"2024-03-16T09:39:56.184208+0000","last_unstale":"2024-03-16T09:40:58.861161+0000","last_undegraded":"2024-03-16T09:40:58.861161+0000","last_fullsized":"2024-03-16T09:40:58.861161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:14.025625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860098+0000","last_change":"2024-03-16T09:39:56.184331+0000","last_active":"2024-03-16T09:40:58.860098+0000","last_peered":"2024-03-16T09:40:58.860098+0000","last_clean":"2024-03-16T09:40:58.860098+0000","last_became_active":"2024-03-16T09:39:56.183919+0000","last_became_peered":"2024-03-16T09:39:56.183919+0000","last_unstale":"2024-03-16T09:40:58.860098+0000","last_undegraded":"2024-03-16T09:40:58.860098+0000","last_fullsized":"2024-03-16T09:40:58.860098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:59.692854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865484+0000","last_change":"2024-03-16T09:39:56.185607+0000","last_active":"2024-03-16T09:40:58.865484+0000","last_peered":"2024-03-16T09:40:58.865484+0000","last_clean":"2024-03-16T09:40:58.865484+0000","last_became_active":"2024-03-16T09:39:56.185278+0000","last_became_peered":"2024-03-16T09:39:56.185278+0000","last_unstale":"2024-03-16T09:40:58.865484+0000","last_undegraded":"2024-03-16T09:40:58.865484+0000","last_fullsized":"2024-03-16T09:40:58.865484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:20.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860105+0000","last_change":"2024-03-16T09:39:55.180852+0000","last_active":"2024-03-16T09:40:58.860105+0000","last_peered":"2024-03-16T09:40:58.860105+0000","last_clean":"2024-03-16T09:40:58.860105+0000","last_became_active":"2024-03-16T09:39:55.180496+0000","last_became_peered":"2024-03-16T09:39:55.180496+0000","last_unstale":"2024-03-16T09:40:58.860105+0000","last_undegraded":"2024-03-16T09:40:58.860105+0000","last_fullsized":"2024-03-16T09:40:58.860105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:15:28.548458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871431+0000","last_change":"2024-03-16T09:39:56.190110+0000","last_active":"2024-03-16T09:40:58.871431+0000","last_peered":"2024-03-16T09:40:58.871431+0000","last_clean":"2024-03-16T09:40:58.871431+0000","last_became_active":"2024-03-16T09:39:56.189711+0000","last_became_peered":"2024-03-16T09:39:56.189711+0000","last_unstale":"2024-03-16T09:40:58.871431+0000","last_undegraded":"2024-03-16T09:40:58.871431+0000","last_fullsized":"2024-03-16T09:40:58.871431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:17.353457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871621+0000","last_change":"2024-03-16T09:39:55.171142+0000","last_active":"2024-03-16T09:40:58.871621+0000","last_peered":"2024-03-16T09:40:58.871621+0000","last_clean":"2024-03-16T09:40:58.871621+0000","last_became_active":"2024-03-16T09:39:55.170793+0000","last_became_peered":"2024-03-16T09:39:55.170793+0000","last_unstale":"2024-03-16T09:40:58.871621+0000","last_undegraded":"2024-03-16T09:40:58.871621+0000","last_fullsized":"2024-03-16T09:40:58.871621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:36.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865733+0000","last_change":"2024-03-16T09:39:56.186802+0000","last_active":"2024-03-16T09:40:58.865733+0000","last_peered":"2024-03-16T09:40:58.865733+0000","last_clean":"2024-03-16T09:40:58.865733+0000","last_became_active":"2024-03-16T09:39:56.186398+0000","last_became_peered":"2024-03-16T09:39:56.186398+0000","last_unstale":"2024-03-16T09:40:58.865733+0000","last_undegraded":"2024-03-16T09:40:58.865733+0000","last_fullsized":"2024-03-16T09:40:58.865733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:05:56.209864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864171+0000","last_change":"2024-03-16T09:39:55.181894+0000","last_active":"2024-03-16T09:40:58.864171+0000","last_peered":"2024-03-16T09:40:58.864171+0000","last_clean":"2024-03-16T09:40:58.864171+0000","last_became_active":"2024-03-16T09:39:55.179434+0000","last_became_peered":"2024-03-16T09:39:55.179434+0000","last_unstale":"2024-03-16T09:40:58.864171+0000","last_undegraded":"2024-03-16T09:40:58.864171+0000","last_fullsized":"2024-03-16T09:40:58.864171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:16.153928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.862941+0000","last_change":"2024-03-16T09:39:56.188464+0000","last_active":"2024-03-16T09:40:58.862941+0000","last_peered":"2024-03-16T09:40:58.862941+0000","last_clean":"2024-03-16T09:40:58.862941+0000","last_became_active":"2024-03-16T09:39:56.188125+0000","last_became_peered":"2024-03-16T09:39:56.188125+0000","last_unstale":"2024-03-16T09:40:58.862941+0000","last_undegraded":"2024-03-16T09:40:58.862941+0000","last_fullsized":"2024-03-16T09:40:58.862941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:46:49.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863856+0000","last_change":"2024-03-16T09:39:55.179379+0000","last_active":"2024-03-16T09:40:58.863856+0000","last_peered":"2024-03-16T09:40:58.863856+0000","last_clean":"2024-03-16T09:40:58.863856+0000","last_became_active":"2024-03-16T09:39:55.178755+0000","last_became_peered":"2024-03-16T09:39:55.178755+0000","last_unstale":"2024-03-16T09:40:58.863856+0000","last_undegraded":"2024-03-16T09:40:58.863856+0000","last_fullsized":"2024-03-16T09:40:58.863856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:07.131417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863602+0000","last_change":"2024-03-16T09:39:56.196655+0000","last_active":"2024-03-16T09:40:58.863602+0000","last_peered":"2024-03-16T09:40:58.863602+0000","last_clean":"2024-03-16T09:40:58.863602+0000","last_became_active":"2024-03-16T09:39:56.196344+0000","last_became_peered":"2024-03-16T09:39:56.196344+0000","last_unstale":"2024-03-16T09:40:58.863602+0000","last_undegraded":"2024-03-16T09:40:58.863602+0000","last_fullsized":"2024-03-16T09:40:58.863602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:12:03.920319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:05.804950+0000","last_change":"2024-03-16T09:39:55.182012+0000","last_active":"2024-03-16T09:41:05.804950+0000","last_peered":"2024-03-16T09:41:05.804950+0000","last_clean":"2024-03-16T09:41:05.804950+0000","last_became_active":"2024-03-16T09:39:55.178651+0000","last_became_peered":"2024-03-16T09:39:55.178651+0000","last_unstale":"2024-03-16T09:41:05.804950+0000","last_undegraded":"2024-03-16T09:41:05.804950+0000","last_fullsized":"2024-03-16T09:41:05.804950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:50:19.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863596+0000","last_change":"2024-03-16T09:39:56.179645+0000","last_active":"2024-03-16T09:40:58.863596+0000","last_peered":"2024-03-16T09:40:58.863596+0000","last_clean":"2024-03-16T09:40:58.863596+0000","last_became_active":"2024-03-16T09:39:56.179257+0000","last_became_peered":"2024-03-16T09:39:56.179257+0000","last_unstale":"2024-03-16T09:40:58.863596+0000","last_undegraded":"2024-03-16T09:40:58.863596+0000","last_fullsized":"2024-03-16T09:40:58.863596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:30:09.104630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.218444+0000","last_change":"2024-03-16T09:39:55.178257+0000","last_active":"2024-03-16T09:40:59.218444+0000","last_peered":"2024-03-16T09:40:59.218444+0000","last_clean":"2024-03-16T09:40:59.218444+0000","last_became_active":"2024-03-16T09:39:55.177984+0000","last_became_peered":"2024-03-16T09:39:55.177984+0000","last_unstale":"2024-03-16T09:40:59.218444+0000","last_undegraded":"2024-03-16T09:40:59.218444+0000","last_fullsized":"2024-03-16T09:40:59.218444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.385624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863340+0000","last_change":"2024-03-16T09:39:56.196821+0000","last_active":"2024-03-16T09:40:58.863340+0000","last_peered":"2024-03-16T09:40:58.863340+0000","last_clean":"2024-03-16T09:40:58.863340+0000","last_became_active":"2024-03-16T09:39:56.196522+0000","last_became_peered":"2024-03-16T09:39:56.196522+0000","last_unstale":"2024-03-16T09:40:58.863340+0000","last_undegraded":"2024-03-16T09:40:58.863340+0000","last_fullsized":"2024-03-16T09:40:58.863340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:31.103530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865998+0000","last_change":"2024-03-16T09:39:55.170290+0000","last_active":"2024-03-16T09:40:58.865998+0000","last_peered":"2024-03-16T09:40:58.865998+0000","last_clean":"2024-03-16T09:40:58.865998+0000","last_became_active":"2024-03-16T09:39:55.169956+0000","last_became_peered":"2024-03-16T09:39:55.169956+0000","last_unstale":"2024-03-16T09:40:58.865998+0000","last_undegraded":"2024-03-16T09:40:58.865998+0000","last_fullsized":"2024-03-16T09:40:58.865998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:01:31.606917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871481+0000","last_change":"2024-03-16T09:39:56.189155+0000","last_active":"2024-03-16T09:40:58.871481+0000","last_peered":"2024-03-16T09:40:58.871481+0000","last_clean":"2024-03-16T09:40:58.871481+0000","last_became_active":"2024-03-16T09:39:56.188827+0000","last_became_peered":"2024-03-16T09:39:56.188827+0000","last_unstale":"2024-03-16T09:40:58.871481+0000","last_undegraded":"2024-03-16T09:40:58.871481+0000","last_fullsized":"2024-03-16T09:40:58.871481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:54.370754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863093+0000","last_change":"2024-03-16T09:39:55.181886+0000","last_active":"2024-03-16T09:40:58.863093+0000","last_peered":"2024-03-16T09:40:58.863093+0000","last_clean":"2024-03-16T09:40:58.863093+0000","last_became_active":"2024-03-16T09:39:55.179836+0000","last_became_peered":"2024-03-16T09:39:55.179836+0000","last_unstale":"2024-03-16T09:40:58.863093+0000","last_undegraded":"2024-03-16T09:40:58.863093+0000","last_fullsized":"2024-03-16T09:40:58.863093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:52:38.772861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.859838+0000","last_change":"2024-03-16T09:39:56.185892+0000","last_active":"2024-03-16T09:40:58.859838+0000","last_peered":"2024-03-16T09:40:58.859838+0000","last_clean":"2024-03-16T09:40:58.859838+0000","last_became_active":"2024-03-16T09:39:56.185459+0000","last_became_peered":"2024-03-16T09:39:56.185459+0000","last_unstale":"2024-03-16T09:40:58.859838+0000","last_undegraded":"2024-03-16T09:40:58.859838+0000","last_fullsized":"2024-03-16T09:40:58.859838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:10.421054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:03.802753+0000","last_change":"2024-03-16T09:39:55.177607+0000","last_active":"2024-03-16T09:41:03.802753+0000","last_peered":"2024-03-16T09:41:03.802753+0000","last_clean":"2024-03-16T09:41:03.802753+0000","last_became_active":"2024-03-16T09:39:55.177265+0000","last_became_peered":"2024-03-16T09:39:55.177265+0000","last_unstale":"2024-03-16T09:41:03.802753+0000","last_undegraded":"2024-03-16T09:41:03.802753+0000","last_fullsized":"2024-03-16T09:41:03.802753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:47.849041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863281+0000","last_change":"2024-03-16T09:39:56.189172+0000","last_active":"2024-03-16T09:40:58.863281+0000","last_peered":"2024-03-16T09:40:58.863281+0000","last_clean":"2024-03-16T09:40:58.863281+0000","last_became_active":"2024-03-16T09:39:56.188798+0000","last_became_peered":"2024-03-16T09:39:56.188798+0000","last_unstale":"2024-03-16T09:40:58.863281+0000","last_undegraded":"2024-03-16T09:40:58.863281+0000","last_fullsized":"2024-03-16T09:40:58.863281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:02:11.119293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860780+0000","last_change":"2024-03-16T09:39:55.180369+0000","last_active":"2024-03-16T09:40:58.860780+0000","last_peered":"2024-03-16T09:40:58.860780+0000","last_clean":"2024-03-16T09:40:58.860780+0000","last_became_active":"2024-03-16T09:39:55.180053+0000","last_became_peered":"2024-03-16T09:39:55.180053+0000","last_unstale":"2024-03-16T09:40:58.860780+0000","last_undegraded":"2024-03-16T09:40:58.860780+0000","last_fullsized":"2024-03-16T09:40:58.860780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:44.990028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865703+0000","last_change":"2024-03-16T09:39:56.198454+0000","last_active":"2024-03-16T09:40:58.865703+0000","last_peered":"2024-03-16T09:40:58.865703+0000","last_clean":"2024-03-16T09:40:58.865703+0000","last_became_active":"2024-03-16T09:39:56.198097+0000","last_became_peered":"2024-03-16T09:39:56.198097+0000","last_unstale":"2024-03-16T09:40:58.865703+0000","last_undegraded":"2024-03-16T09:40:58.865703+0000","last_fullsized":"2024-03-16T09:40:58.865703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:12.494363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863700+0000","last_change":"2024-03-16T09:39:55.174221+0000","last_active":"2024-03-16T09:40:58.863700+0000","last_peered":"2024-03-16T09:40:58.863700+0000","last_clean":"2024-03-16T09:40:58.863700+0000","last_became_active":"2024-03-16T09:39:55.173920+0000","last_became_peered":"2024-03-16T09:39:55.173920+0000","last_unstale":"2024-03-16T09:40:58.863700+0000","last_undegraded":"2024-03-16T09:40:58.863700+0000","last_fullsized":"2024-03-16T09:40:58.863700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:25.108414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863385+0000","last_change":"2024-03-16T09:39:56.196849+0000","last_active":"2024-03-16T09:40:58.863385+0000","last_peered":"2024-03-16T09:40:58.863385+0000","last_clean":"2024-03-16T09:40:58.863385+0000","last_became_active":"2024-03-16T09:39:56.196515+0000","last_became_peered":"2024-03-16T09:39:56.196515+0000","last_unstale":"2024-03-16T09:40:58.863385+0000","last_undegraded":"2024-03-16T09:40:58.863385+0000","last_fullsized":"2024-03-16T09:40:58.863385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:03.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'29","reported_seq":188,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:04.803597+0000","last_change":"2024-03-16T09:39:55.177788+0000","last_active":"2024-03-16T09:41:04.803597+0000","last_peered":"2024-03-16T09:41:04.803597+0000","last_clean":"2024-03-16T09:41:04.803597+0000","last_became_active":"2024-03-16T09:39:55.177443+0000","last_became_peered":"2024-03-16T09:39:55.177443+0000","last_unstale":"2024-03-16T09:41:04.803597+0000","last_undegraded":"2024-03-16T09:41:04.803597+0000","last_fullsized":"2024-03-16T09:41:04.803597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:39.221712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":140,"num_read_kb":128,"num_write":30,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863958+0000","last_change":"2024-03-16T09:39:56.178976+0000","last_active":"2024-03-16T09:40:58.863958+0000","last_peered":"2024-03-16T09:40:58.863958+0000","last_clean":"2024-03-16T09:40:58.863958+0000","last_became_active":"2024-03-16T09:39:56.178455+0000","last_became_peered":"2024-03-16T09:39:56.178455+0000","last_unstale":"2024-03-16T09:40:58.863958+0000","last_undegraded":"2024-03-16T09:40:58.863958+0000","last_fullsized":"2024-03-16T09:40:58.863958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:52:08.064792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865191+0000","last_change":"2024-03-16T09:39:55.182579+0000","last_active":"2024-03-16T09:40:58.865191+0000","last_peered":"2024-03-16T09:40:58.865191+0000","last_clean":"2024-03-16T09:40:58.865191+0000","last_became_active":"2024-03-16T09:39:55.181298+0000","last_became_peered":"2024-03-16T09:39:55.181298+0000","last_unstale":"2024-03-16T09:40:58.865191+0000","last_undegraded":"2024-03-16T09:40:58.865191+0000","last_fullsized":"2024-03-16T09:40:58.865191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:35.768812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866091+0000","last_change":"2024-03-16T09:39:56.188519+0000","last_active":"2024-03-16T09:40:58.866091+0000","last_peered":"2024-03-16T09:40:58.866091+0000","last_clean":"2024-03-16T09:40:58.866091+0000","last_became_active":"2024-03-16T09:39:56.188031+0000","last_became_peered":"2024-03-16T09:39:56.188031+0000","last_unstale":"2024-03-16T09:40:58.866091+0000","last_undegraded":"2024-03-16T09:40:58.866091+0000","last_fullsized":"2024-03-16T09:40:58.866091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:20:31.765657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864304+0000","last_change":"2024-03-16T09:39:55.177523+0000","last_active":"2024-03-16T09:40:58.864304+0000","last_peered":"2024-03-16T09:40:58.864304+0000","last_clean":"2024-03-16T09:40:58.864304+0000","last_became_active":"2024-03-16T09:39:55.177143+0000","last_became_peered":"2024-03-16T09:39:55.177143+0000","last_unstale":"2024-03-16T09:40:58.864304+0000","last_undegraded":"2024-03-16T09:40:58.864304+0000","last_fullsized":"2024-03-16T09:40:58.864304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:05.357611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.867151+0000","last_change":"2024-03-16T09:40:58.867151+0000","last_active":"2024-03-16T09:40:58.867151+0000","last_peered":"2024-03-16T09:40:58.867151+0000","last_clean":"2024-03-16T09:40:58.867151+0000","last_became_active":"2024-03-16T09:40:58.866756+0000","last_became_peered":"2024-03-16T09:40:58.866756+0000","last_unstale":"2024-03-16T09:40:58.867151+0000","last_undegraded":"2024-03-16T09:40:58.867151+0000","last_fullsized":"2024-03-16T09:40:58.867151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:53.323964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:02.811737+0000","last_change":"2024-03-16T09:39:55.175419+0000","last_active":"2024-03-16T09:41:02.811737+0000","last_peered":"2024-03-16T09:41:02.811737+0000","last_clean":"2024-03-16T09:41:02.811737+0000","last_became_active":"2024-03-16T09:39:55.175126+0000","last_became_peered":"2024-03-16T09:39:55.175126+0000","last_unstale":"2024-03-16T09:41:02.811737+0000","last_undegraded":"2024-03-16T09:41:02.811737+0000","last_fullsized":"2024-03-16T09:41:02.811737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:11.696911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860379+0000","last_change":"2024-03-16T09:39:56.176714+0000","last_active":"2024-03-16T09:40:58.860379+0000","last_peered":"2024-03-16T09:40:58.860379+0000","last_clean":"2024-03-16T09:40:58.860379+0000","last_became_active":"2024-03-16T09:39:56.176245+0000","last_became_peered":"2024-03-16T09:39:56.176245+0000","last_unstale":"2024-03-16T09:40:58.860379+0000","last_undegraded":"2024-03-16T09:40:58.860379+0000","last_fullsized":"2024-03-16T09:40:58.860379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:32.430207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863725+0000","last_change":"2024-03-16T09:39:55.181486+0000","last_active":"2024-03-16T09:40:58.863725+0000","last_peered":"2024-03-16T09:40:58.863725+0000","last_clean":"2024-03-16T09:40:58.863725+0000","last_became_active":"2024-03-16T09:39:55.180128+0000","last_became_peered":"2024-03-16T09:39:55.180128+0000","last_unstale":"2024-03-16T09:40:58.863725+0000","last_undegraded":"2024-03-16T09:40:58.863725+0000","last_fullsized":"2024-03-16T09:40:58.863725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:45.323870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864169+0000","last_change":"2024-03-16T09:39:56.196692+0000","last_active":"2024-03-16T09:40:58.864169+0000","last_peered":"2024-03-16T09:40:58.864169+0000","last_clean":"2024-03-16T09:40:58.864169+0000","last_became_active":"2024-03-16T09:39:56.196403+0000","last_became_peered":"2024-03-16T09:39:56.196403+0000","last_unstale":"2024-03-16T09:40:58.864169+0000","last_undegraded":"2024-03-16T09:40:58.864169+0000","last_fullsized":"2024-03-16T09:40:58.864169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:00:29.426476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871091+0000","last_change":"2024-03-16T09:40:58.870587+0000","last_active":"2024-03-16T09:40:58.871091+0000","last_peered":"2024-03-16T09:40:58.871091+0000","last_clean":"2024-03-16T09:40:58.871091+0000","last_became_active":"2024-03-16T09:40:58.864977+0000","last_became_peered":"2024-03-16T09:40:58.864977+0000","last_unstale":"2024-03-16T09:40:58.871091+0000","last_undegraded":"2024-03-16T09:40:58.871091+0000","last_fullsized":"2024-03-16T09:40:58.871091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:18.189307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866074+0000","last_change":"2024-03-16T09:40:58.866074+0000","last_active":"2024-03-16T09:40:58.866074+0000","last_peered":"2024-03-16T09:40:58.866074+0000","last_clean":"2024-03-16T09:40:58.866074+0000","last_became_active":"2024-03-16T09:40:58.865711+0000","last_became_peered":"2024-03-16T09:40:58.865711+0000","last_unstale":"2024-03-16T09:40:58.866074+0000","last_undegraded":"2024-03-16T09:40:58.866074+0000","last_fullsized":"2024-03-16T09:40:58.866074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:57.137802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.859988+0000","last_change":"2024-03-16T09:39:55.175666+0000","last_active":"2024-03-16T09:40:58.859988+0000","last_peered":"2024-03-16T09:40:58.859988+0000","last_clean":"2024-03-16T09:40:58.859988+0000","last_became_active":"2024-03-16T09:39:55.175337+0000","last_became_peered":"2024-03-16T09:39:55.175337+0000","last_unstale":"2024-03-16T09:40:58.859988+0000","last_undegraded":"2024-03-16T09:40:58.859988+0000","last_fullsized":"2024-03-16T09:40:58.859988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:45:13.935669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.219039+0000","last_change":"2024-03-16T09:39:56.184878+0000","last_active":"2024-03-16T09:40:59.219039+0000","last_peered":"2024-03-16T09:40:59.219039+0000","last_clean":"2024-03-16T09:40:59.219039+0000","last_became_active":"2024-03-16T09:39:56.184480+0000","last_became_peered":"2024-03-16T09:39:56.184480+0000","last_unstale":"2024-03-16T09:40:59.219039+0000","last_undegraded":"2024-03-16T09:40:59.219039+0000","last_fullsized":"2024-03-16T09:40:59.219039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:40:01.286993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860584+0000","last_change":"2024-03-16T09:39:55.179189+0000","last_active":"2024-03-16T09:40:58.860584+0000","last_peered":"2024-03-16T09:40:58.860584+0000","last_clean":"2024-03-16T09:40:58.860584+0000","last_became_active":"2024-03-16T09:39:55.178859+0000","last_became_peered":"2024-03-16T09:39:55.178859+0000","last_unstale":"2024-03-16T09:40:58.860584+0000","last_undegraded":"2024-03-16T09:40:58.860584+0000","last_fullsized":"2024-03-16T09:40:58.860584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:54:15.498386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.219025+0000","last_change":"2024-03-16T09:39:56.200665+0000","last_active":"2024-03-16T09:40:59.219025+0000","last_peered":"2024-03-16T09:40:59.219025+0000","last_clean":"2024-03-16T09:40:59.219025+0000","last_became_active":"2024-03-16T09:39:56.200326+0000","last_became_peered":"2024-03-16T09:39:56.200326+0000","last_unstale":"2024-03-16T09:40:59.219025+0000","last_undegraded":"2024-03-16T09:40:59.219025+0000","last_fullsized":"2024-03-16T09:40:59.219025+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:34.778630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863128+0000","last_change":"2024-03-16T09:39:56.182670+0000","last_active":"2024-03-16T09:40:58.863128+0000","last_peered":"2024-03-16T09:40:58.863128+0000","last_clean":"2024-03-16T09:40:58.863128+0000","last_became_active":"2024-03-16T09:39:56.182009+0000","last_became_peered":"2024-03-16T09:39:56.182009+0000","last_unstale":"2024-03-16T09:40:58.863128+0000","last_undegraded":"2024-03-16T09:40:58.863128+0000","last_fullsized":"2024-03-16T09:40:58.863128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:28:03.106071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:04.813710+0000","last_change":"2024-03-16T09:39:55.171609+0000","last_active":"2024-03-16T09:41:04.813710+0000","last_peered":"2024-03-16T09:41:04.813710+0000","last_clean":"2024-03-16T09:41:04.813710+0000","last_became_active":"2024-03-16T09:39:55.171301+0000","last_became_peered":"2024-03-16T09:39:55.171301+0000","last_unstale":"2024-03-16T09:41:04.813710+0000","last_undegraded":"2024-03-16T09:41:04.813710+0000","last_fullsized":"2024-03-16T09:41:04.813710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:28.265342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.867266+0000","last_change":"2024-03-16T09:40:58.865311+0000","last_active":"2024-03-16T09:40:58.867266+0000","last_peered":"2024-03-16T09:40:58.867266+0000","last_clean":"2024-03-16T09:40:58.867266+0000","last_became_active":"2024-03-16T09:40:58.864918+0000","last_became_peered":"2024-03-16T09:40:58.864918+0000","last_unstale":"2024-03-16T09:40:58.867266+0000","last_undegraded":"2024-03-16T09:40:58.867266+0000","last_fullsized":"2024-03-16T09:40:58.867266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863236+0000","last_change":"2024-03-16T09:39:55.172824+0000","last_active":"2024-03-16T09:40:58.863236+0000","last_peered":"2024-03-16T09:40:58.863236+0000","last_clean":"2024-03-16T09:40:58.863236+0000","last_became_active":"2024-03-16T09:39:55.172493+0000","last_became_peered":"2024-03-16T09:39:55.172493+0000","last_unstale":"2024-03-16T09:40:58.863236+0000","last_undegraded":"2024-03-16T09:40:58.863236+0000","last_fullsized":"2024-03-16T09:40:58.863236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:39.188176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860453+0000","last_change":"2024-03-16T09:38:43.693644+0000","last_active":"2024-03-16T09:40:58.860453+0000","last_peered":"2024-03-16T09:40:58.860453+0000","last_clean":"2024-03-16T09:40:58.860453+0000","last_became_active":"2024-03-16T09:38:43.693202+0000","last_became_peered":"2024-03-16T09:38:43.693202+0000","last_unstale":"2024-03-16T09:40:58.860453+0000","last_undegraded":"2024-03-16T09:40:58.860453+0000","last_fullsized":"2024-03-16T09:40:58.860453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_clean_scrub_stamp":"2024-03-16T09:35:24.153508+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:43:06.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865878+0000","last_change":"2024-03-16T09:39:56.197109+0000","last_active":"2024-03-16T09:40:58.865878+0000","last_peered":"2024-03-16T09:40:58.865878+0000","last_clean":"2024-03-16T09:40:58.865878+0000","last_became_active":"2024-03-16T09:39:56.196775+0000","last_became_peered":"2024-03-16T09:39:56.196775+0000","last_unstale":"2024-03-16T09:40:58.865878+0000","last_undegraded":"2024-03-16T09:40:58.865878+0000","last_fullsized":"2024-03-16T09:40:58.865878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:29:10.401423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860552+0000","last_change":"2024-03-16T09:39:55.175744+0000","last_active":"2024-03-16T09:40:58.860552+0000","last_peered":"2024-03-16T09:40:58.860552+0000","last_clean":"2024-03-16T09:40:58.860552+0000","last_became_active":"2024-03-16T09:39:55.175445+0000","last_became_peered":"2024-03-16T09:39:55.175445+0000","last_unstale":"2024-03-16T09:40:58.860552+0000","last_undegraded":"2024-03-16T09:40:58.860552+0000","last_fullsized":"2024-03-16T09:40:58.860552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:39.494340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864068+0000","last_change":"2024-03-16T09:39:55.183441+0000","last_active":"2024-03-16T09:40:58.864068+0000","last_peered":"2024-03-16T09:40:58.864068+0000","last_clean":"2024-03-16T09:40:58.864068+0000","last_became_active":"2024-03-16T09:39:55.183115+0000","last_became_peered":"2024-03-16T09:39:55.183115+0000","last_unstale":"2024-03-16T09:40:58.864068+0000","last_undegraded":"2024-03-16T09:40:58.864068+0000","last_fullsized":"2024-03-16T09:40:58.864068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:55:31.525201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866116+0000","last_change":"2024-03-16T09:39:56.179205+0000","last_active":"2024-03-16T09:40:58.866116+0000","last_peered":"2024-03-16T09:40:58.866116+0000","last_clean":"2024-03-16T09:40:58.866116+0000","last_became_active":"2024-03-16T09:39:56.178849+0000","last_became_peered":"2024-03-16T09:39:56.178849+0000","last_unstale":"2024-03-16T09:40:58.866116+0000","last_undegraded":"2024-03-16T09:40:58.866116+0000","last_fullsized":"2024-03-16T09:40:58.866116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:10.016099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860157+0000","last_change":"2024-03-16T09:39:55.180859+0000","last_active":"2024-03-16T09:40:58.860157+0000","last_peered":"2024-03-16T09:40:58.860157+0000","last_clean":"2024-03-16T09:40:58.860157+0000","last_became_active":"2024-03-16T09:39:55.180514+0000","last_became_peered":"2024-03-16T09:39:55.180514+0000","last_unstale":"2024-03-16T09:40:58.860157+0000","last_undegraded":"2024-03-16T09:40:58.860157+0000","last_fullsized":"2024-03-16T09:40:58.860157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:46.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871538+0000","last_change":"2024-03-16T09:39:56.187626+0000","last_active":"2024-03-16T09:40:58.871538+0000","last_peered":"2024-03-16T09:40:58.871538+0000","last_clean":"2024-03-16T09:40:58.871538+0000","last_became_active":"2024-03-16T09:39:56.187284+0000","last_became_peered":"2024-03-16T09:39:56.187284+0000","last_unstale":"2024-03-16T09:40:58.871538+0000","last_undegraded":"2024-03-16T09:40:58.871538+0000","last_fullsized":"2024-03-16T09:40:58.871538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:23.511603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864099+0000","last_change":"2024-03-16T09:39:55.172912+0000","last_active":"2024-03-16T09:40:58.864099+0000","last_peered":"2024-03-16T09:40:58.864099+0000","last_clean":"2024-03-16T09:40:58.864099+0000","last_became_active":"2024-03-16T09:39:55.172592+0000","last_became_peered":"2024-03-16T09:39:55.172592+0000","last_unstale":"2024-03-16T09:40:58.864099+0000","last_undegraded":"2024-03-16T09:40:58.864099+0000","last_fullsized":"2024-03-16T09:40:58.864099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:24.344588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865663+0000","last_change":"2024-03-16T09:39:56.197652+0000","last_active":"2024-03-16T09:40:58.865663+0000","last_peered":"2024-03-16T09:40:58.865663+0000","last_clean":"2024-03-16T09:40:58.865663+0000","last_became_active":"2024-03-16T09:39:56.197230+0000","last_became_peered":"2024-03-16T09:39:56.197230+0000","last_unstale":"2024-03-16T09:40:58.865663+0000","last_undegraded":"2024-03-16T09:40:58.865663+0000","last_fullsized":"2024-03-16T09:40:58.865663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:45:08.661372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.869780+0000","last_change":"2024-03-16T09:40:58.869780+0000","last_active":"2024-03-16T09:40:58.869780+0000","last_peered":"2024-03-16T09:40:58.869780+0000","last_clean":"2024-03-16T09:40:58.869780+0000","last_became_active":"2024-03-16T09:40:58.866452+0000","last_became_peered":"2024-03-16T09:40:58.866452+0000","last_unstale":"2024-03-16T09:40:58.869780+0000","last_undegraded":"2024-03-16T09:40:58.869780+0000","last_fullsized":"2024-03-16T09:40:58.869780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:07:21.751383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863491+0000","last_change":"2024-03-16T09:39:56.179326+0000","last_active":"2024-03-16T09:40:58.863491+0000","last_peered":"2024-03-16T09:40:58.863491+0000","last_clean":"2024-03-16T09:40:58.863491+0000","last_became_active":"2024-03-16T09:39:56.178681+0000","last_became_peered":"2024-03-16T09:39:56.178681+0000","last_unstale":"2024-03-16T09:40:58.863491+0000","last_undegraded":"2024-03-16T09:40:58.863491+0000","last_fullsized":"2024-03-16T09:40:58.863491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:26.137807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871338+0000","last_change":"2024-03-16T09:39:55.173820+0000","last_active":"2024-03-16T09:40:58.871338+0000","last_peered":"2024-03-16T09:40:58.871338+0000","last_clean":"2024-03-16T09:40:58.871338+0000","last_became_active":"2024-03-16T09:39:55.173472+0000","last_became_peered":"2024-03-16T09:39:55.173472+0000","last_unstale":"2024-03-16T09:40:58.871338+0000","last_undegraded":"2024-03-16T09:40:58.871338+0000","last_fullsized":"2024-03-16T09:40:58.871338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:14.082367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860822+0000","last_change":"2024-03-16T09:39:56.178535+0000","last_active":"2024-03-16T09:40:58.860822+0000","last_peered":"2024-03-16T09:40:58.860822+0000","last_clean":"2024-03-16T09:40:58.860822+0000","last_became_active":"2024-03-16T09:39:56.178210+0000","last_became_peered":"2024-03-16T09:39:56.178210+0000","last_unstale":"2024-03-16T09:40:58.860822+0000","last_undegraded":"2024-03-16T09:40:58.860822+0000","last_fullsized":"2024-03-16T09:40:58.860822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:18.680469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864920+0000","last_change":"2024-03-16T09:39:55.183074+0000","last_active":"2024-03-16T09:40:58.864920+0000","last_peered":"2024-03-16T09:40:58.864920+0000","last_clean":"2024-03-16T09:40:58.864920+0000","last_became_active":"2024-03-16T09:39:55.182695+0000","last_became_peered":"2024-03-16T09:39:55.182695+0000","last_unstale":"2024-03-16T09:40:58.864920+0000","last_undegraded":"2024-03-16T09:40:58.864920+0000","last_fullsized":"2024-03-16T09:40:58.864920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:51.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871544+0000","last_change":"2024-03-16T09:39:56.193434+0000","last_active":"2024-03-16T09:40:58.871544+0000","last_peered":"2024-03-16T09:40:58.871544+0000","last_clean":"2024-03-16T09:40:58.871544+0000","last_became_active":"2024-03-16T09:39:56.193051+0000","last_became_peered":"2024-03-16T09:39:56.193051+0000","last_unstale":"2024-03-16T09:40:58.871544+0000","last_undegraded":"2024-03-16T09:40:58.871544+0000","last_fullsized":"2024-03-16T09:40:58.871544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:12:26.767117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863710+0000","last_change":"2024-03-16T09:39:55.179916+0000","last_active":"2024-03-16T09:40:58.863710+0000","last_peered":"2024-03-16T09:40:58.863710+0000","last_clean":"2024-03-16T09:40:58.863710+0000","last_became_active":"2024-03-16T09:39:55.179635+0000","last_became_peered":"2024-03-16T09:39:55.179635+0000","last_unstale":"2024-03-16T09:40:58.863710+0000","last_undegraded":"2024-03-16T09:40:58.863710+0000","last_fullsized":"2024-03-16T09:40:58.863710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:08.955039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860659+0000","last_change":"2024-03-16T09:39:56.194044+0000","last_active":"2024-03-16T09:40:58.860659+0000","last_peered":"2024-03-16T09:40:58.860659+0000","last_clean":"2024-03-16T09:40:58.860659+0000","last_became_active":"2024-03-16T09:39:56.193490+0000","last_became_peered":"2024-03-16T09:39:56.193490+0000","last_unstale":"2024-03-16T09:40:58.860659+0000","last_undegraded":"2024-03-16T09:40:58.860659+0000","last_fullsized":"2024-03-16T09:40:58.860659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:19:59.916892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865495+0000","last_change":"2024-03-16T09:39:55.182652+0000","last_active":"2024-03-16T09:40:58.865495+0000","last_peered":"2024-03-16T09:40:58.865495+0000","last_clean":"2024-03-16T09:40:58.865495+0000","last_became_active":"2024-03-16T09:39:55.181432+0000","last_became_peered":"2024-03-16T09:39:55.181432+0000","last_unstale":"2024-03-16T09:40:58.865495+0000","last_undegraded":"2024-03-16T09:40:58.865495+0000","last_fullsized":"2024-03-16T09:40:58.865495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:34.959845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860880+0000","last_change":"2024-03-16T09:39:56.188970+0000","last_active":"2024-03-16T09:40:58.860880+0000","last_peered":"2024-03-16T09:40:58.860880+0000","last_clean":"2024-03-16T09:40:58.860880+0000","last_became_active":"2024-03-16T09:39:56.188607+0000","last_became_peered":"2024-03-16T09:39:56.188607+0000","last_unstale":"2024-03-16T09:40:58.860880+0000","last_undegraded":"2024-03-16T09:40:58.860880+0000","last_fullsized":"2024-03-16T09:40:58.860880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:39:24.532114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863497+0000","last_change":"2024-03-16T09:39:55.181908+0000","last_active":"2024-03-16T09:40:58.863497+0000","last_peered":"2024-03-16T09:40:58.863497+0000","last_clean":"2024-03-16T09:40:58.863497+0000","last_became_active":"2024-03-16T09:39:55.181055+0000","last_became_peered":"2024-03-16T09:39:55.181055+0000","last_unstale":"2024-03-16T09:40:58.863497+0000","last_undegraded":"2024-03-16T09:40:58.863497+0000","last_fullsized":"2024-03-16T09:40:58.863497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:38.256664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860744+0000","last_change":"2024-03-16T09:39:56.189743+0000","last_active":"2024-03-16T09:40:58.860744+0000","last_peered":"2024-03-16T09:40:58.860744+0000","last_clean":"2024-03-16T09:40:58.860744+0000","last_became_active":"2024-03-16T09:39:56.189339+0000","last_became_peered":"2024-03-16T09:39:56.189339+0000","last_unstale":"2024-03-16T09:40:58.860744+0000","last_undegraded":"2024-03-16T09:40:58.860744+0000","last_fullsized":"2024-03-16T09:40:58.860744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:54.694191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.874651+0000","last_change":"2024-03-16T09:40:58.874651+0000","last_active":"2024-03-16T09:40:58.874651+0000","last_peered":"2024-03-16T09:40:58.874651+0000","last_clean":"2024-03-16T09:40:58.874651+0000","last_became_active":"2024-03-16T09:40:58.872329+0000","last_became_peered":"2024-03-16T09:40:58.872329+0000","last_unstale":"2024-03-16T09:40:58.874651+0000","last_undegraded":"2024-03-16T09:40:58.874651+0000","last_fullsized":"2024-03-16T09:40:58.874651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:05.424197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860934+0000","last_change":"2024-03-16T09:39:56.181636+0000","last_active":"2024-03-16T09:40:58.860934+0000","last_peered":"2024-03-16T09:40:58.860934+0000","last_clean":"2024-03-16T09:40:58.860934+0000","last_became_active":"2024-03-16T09:39:56.181309+0000","last_became_peered":"2024-03-16T09:39:56.181309+0000","last_unstale":"2024-03-16T09:40:58.860934+0000","last_undegraded":"2024-03-16T09:40:58.860934+0000","last_fullsized":"2024-03-16T09:40:58.860934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:36.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860897+0000","last_change":"2024-03-16T09:39:56.186640+0000","last_active":"2024-03-16T09:40:58.860897+0000","last_peered":"2024-03-16T09:40:58.860897+0000","last_clean":"2024-03-16T09:40:58.860897+0000","last_became_active":"2024-03-16T09:39:56.186311+0000","last_became_peered":"2024-03-16T09:39:56.186311+0000","last_unstale":"2024-03-16T09:40:58.860897+0000","last_undegraded":"2024-03-16T09:40:58.860897+0000","last_fullsized":"2024-03-16T09:40:58.860897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:58:44.451068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865167+0000","last_change":"2024-03-16T09:39:56.196887+0000","last_active":"2024-03-16T09:40:58.865167+0000","last_peered":"2024-03-16T09:40:58.865167+0000","last_clean":"2024-03-16T09:40:58.865167+0000","last_became_active":"2024-03-16T09:39:56.196568+0000","last_became_peered":"2024-03-16T09:39:56.196568+0000","last_unstale":"2024-03-16T09:40:58.865167+0000","last_undegraded":"2024-03-16T09:40:58.865167+0000","last_fullsized":"2024-03-16T09:40:58.865167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:08:35.577630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865027+0000","last_change":"2024-03-16T09:39:56.186394+0000","last_active":"2024-03-16T09:40:58.865027+0000","last_peered":"2024-03-16T09:40:58.865027+0000","last_clean":"2024-03-16T09:40:58.865027+0000","last_became_active":"2024-03-16T09:39:56.186053+0000","last_became_peered":"2024-03-16T09:39:56.186053+0000","last_unstale":"2024-03-16T09:40:58.865027+0000","last_undegraded":"2024-03-16T09:40:58.865027+0000","last_fullsized":"2024-03-16T09:40:58.865027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:41.529601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865405+0000","last_change":"2024-03-16T09:39:56.197622+0000","last_active":"2024-03-16T09:40:58.865405+0000","last_peered":"2024-03-16T09:40:58.865405+0000","last_clean":"2024-03-16T09:40:58.865405+0000","last_became_active":"2024-03-16T09:39:56.196885+0000","last_became_peered":"2024-03-16T09:39:56.196885+0000","last_unstale":"2024-03-16T09:40:58.865405+0000","last_undegraded":"2024-03-16T09:40:58.865405+0000","last_fullsized":"2024-03-16T09:40:58.865405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:08:50.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.868417+0000","last_change":"2024-03-16T09:40:58.868417+0000","last_active":"2024-03-16T09:40:58.868417+0000","last_peered":"2024-03-16T09:40:58.868417+0000","last_clean":"2024-03-16T09:40:58.868417+0000","last_became_active":"2024-03-16T09:40:58.868100+0000","last_became_peered":"2024-03-16T09:40:58.868100+0000","last_unstale":"2024-03-16T09:40:58.868417+0000","last_undegraded":"2024-03-16T09:40:58.868417+0000","last_fullsized":"2024-03-16T09:40:58.868417+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:51.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.861123+0000","last_change":"2024-03-16T09:39:56.181262+0000","last_active":"2024-03-16T09:40:58.861123+0000","last_peered":"2024-03-16T09:40:58.861123+0000","last_clean":"2024-03-16T09:40:58.861123+0000","last_became_active":"2024-03-16T09:39:56.180916+0000","last_became_peered":"2024-03-16T09:39:56.180916+0000","last_unstale":"2024-03-16T09:40:58.861123+0000","last_undegraded":"2024-03-16T09:40:58.861123+0000","last_fullsized":"2024-03-16T09:40:58.861123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:34:45.056025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865776+0000","last_change":"2024-03-16T09:39:56.176621+0000","last_active":"2024-03-16T09:40:58.865776+0000","last_peered":"2024-03-16T09:40:58.865776+0000","last_clean":"2024-03-16T09:40:58.865776+0000","last_became_active":"2024-03-16T09:39:56.176296+0000","last_became_peered":"2024-03-16T09:39:56.176296+0000","last_unstale":"2024-03-16T09:40:58.865776+0000","last_undegraded":"2024-03-16T09:40:58.865776+0000","last_fullsized":"2024-03-16T09:40:58.865776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:54.673937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871473+0000","last_change":"2024-03-16T09:39:56.197689+0000","last_active":"2024-03-16T09:40:58.871473+0000","last_peered":"2024-03-16T09:40:58.871473+0000","last_clean":"2024-03-16T09:40:58.871473+0000","last_became_active":"2024-03-16T09:39:56.197343+0000","last_became_peered":"2024-03-16T09:39:56.197343+0000","last_unstale":"2024-03-16T09:40:58.871473+0000","last_undegraded":"2024-03-16T09:40:58.871473+0000","last_fullsized":"2024-03-16T09:40:58.871473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:28:10.218048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.862904+0000","last_change":"2024-03-16T09:39:56.193551+0000","last_active":"2024-03-16T09:40:58.862904+0000","last_peered":"2024-03-16T09:40:58.862904+0000","last_clean":"2024-03-16T09:40:58.862904+0000","last_became_active":"2024-03-16T09:39:56.192901+0000","last_became_peered":"2024-03-16T09:39:56.192901+0000","last_unstale":"2024-03-16T09:40:58.862904+0000","last_undegraded":"2024-03-16T09:40:58.862904+0000","last_fullsized":"2024-03-16T09:40:58.862904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:50:15.528370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871264+0000","last_change":"2024-03-16T09:39:56.188484+0000","last_active":"2024-03-16T09:40:58.871264+0000","last_peered":"2024-03-16T09:40:58.871264+0000","last_clean":"2024-03-16T09:40:58.871264+0000","last_became_active":"2024-03-16T09:39:56.188144+0000","last_became_peered":"2024-03-16T09:39:56.188144+0000","last_unstale":"2024-03-16T09:40:58.871264+0000","last_undegraded":"2024-03-16T09:40:58.871264+0000","last_fullsized":"2024-03-16T09:40:58.871264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:01:10.042765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863200+0000","last_change":"2024-03-16T09:39:56.187448+0000","last_active":"2024-03-16T09:40:58.863200+0000","last_peered":"2024-03-16T09:40:58.863200+0000","last_clean":"2024-03-16T09:40:58.863200+0000","last_became_active":"2024-03-16T09:39:56.186772+0000","last_became_peered":"2024-03-16T09:39:56.186772+0000","last_unstale":"2024-03-16T09:40:58.863200+0000","last_undegraded":"2024-03-16T09:40:58.863200+0000","last_fullsized":"2024-03-16T09:40:58.863200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:45:43.315011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866205+0000","last_change":"2024-03-16T09:39:56.191004+0000","last_active":"2024-03-16T09:40:58.866205+0000","last_peered":"2024-03-16T09:40:58.866205+0000","last_clean":"2024-03-16T09:40:58.866205+0000","last_became_active":"2024-03-16T09:39:56.190646+0000","last_became_peered":"2024-03-16T09:39:56.190646+0000","last_unstale":"2024-03-16T09:40:58.866205+0000","last_undegraded":"2024-03-16T09:40:58.866205+0000","last_fullsized":"2024-03-16T09:40:58.866205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:03.191129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860886+0000","last_change":"2024-03-16T09:39:56.197552+0000","last_active":"2024-03-16T09:40:58.860886+0000","last_peered":"2024-03-16T09:40:58.860886+0000","last_clean":"2024-03-16T09:40:58.860886+0000","last_became_active":"2024-03-16T09:39:56.197215+0000","last_became_peered":"2024-03-16T09:39:56.197215+0000","last_unstale":"2024-03-16T09:40:58.860886+0000","last_undegraded":"2024-03-16T09:40:58.860886+0000","last_fullsized":"2024-03-16T09:40:58.860886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:45.795328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63633,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":423,"num_read_kb":278,"num_write":155,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":460713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612640,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1650688,"data_stored":1112195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776166,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037803,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":460825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1564672,"data_stored":1052386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462963,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":460316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":520132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":461587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":521482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":460627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":460753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36884,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708268,"statfs":{"total":95995035648,"available":95957266432,"internally_reserved":0,"allocated":1576960,"data_stored":1052070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T09:41:09.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T09:41:10.005 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:10.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:10 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:10 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:10 smithi129 ceph-mon[35841]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:10.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:10 smithi129 ceph-mon[35841]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:10.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:10 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:41:10.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:10 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:41:10.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:10 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.349 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.349 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.349 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:10 smithi077 ceph-mon[32171]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:10.349 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:10 smithi077 ceph-mon[32171]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:10.350 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:41:10.350 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:41:10.350 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:10 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:10 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:10.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:10 smithi119 ceph-mon[35632]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:10.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:10 smithi119 ceph-mon[35632]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-16T09:41:10.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:10 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:41:10.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:10 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:41:10.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:10 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 -- 172.21.15.77:0/430427169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 msgr2=0x7f79181086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 --2- 172.21.15.77:0/430427169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181086e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f79080099c0 tx=0x7f7908009cd0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 -- 172.21.15.77:0/430427169 shutdown_connections 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 --2- 172.21.15.77:0/430427169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 0x7f791810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 --2- 172.21.15.77:0/430427169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181086e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 --2- 172.21.15.77:0/430427169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7918102320 0x7f79181027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 -- 172.21.15.77:0/430427169 >> 172.21.15.77:0/430427169 conn(0x7f7918079400 msgr2=0x7f7918079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 -- 172.21.15.77:0/430427169 shutdown_connections 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.593+0000 7f791e70b700 1 -- 172.21.15.77:0/430427169 wait complete. 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 Processor -- start 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 -- start start 2024-03-16T09:41:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7918102320 0x7f791819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 0x7f791819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7918112430 con 0x7f7918108300 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f79181122b0 con 0x7f7918102ce0 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791e70b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79181125b0 con 0x7f7918102320 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f79177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 0x7f791819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f791cca8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.594+0000 7f7917fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7918102320 0x7f791819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f79177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 0x7f791819eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:58716/0 (socket says 172.21.15.77:58716) 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791cca8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45612/0 (socket says 172.21.15.77:45612) 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f79177fe700 1 -- 172.21.15.77:0/4205064397 learned_addr learned my addr 172.21.15.77:0/4205064397 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f7917fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7918102320 0x7f791819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:49808/0 (socket says 172.21.15.77:49808) 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791cca8700 1 -- 172.21.15.77:0/4205064397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7918102320 msgr2=0x7f791819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791cca8700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7918102320 0x7f791819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791cca8700 1 -- 172.21.15.77:0/4205064397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 msgr2=0x7f791819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791cca8700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 0x7f791819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791cca8700 1 -- 172.21.15.77:0/4205064397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7918198fb0 con 0x7f7918108300 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f79177fe700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 0x7f791819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791cca8700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181986d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f791000efc0 tx=0x7f791000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f79157fa700 1 -- 172.21.15.77:0/4205064397 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79100092e0 con 0x7f7918108300 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f79157fa700 1 -- 172.21.15.77:0/4205064397 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7910004bb0 con 0x7f7918108300 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7918199240 con 0x7f7918108300 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f79157fa700 1 -- 172.21.15.77:0/4205064397 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910004020 con 0x7f7918108300 2024-03-16T09:41:11.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.595+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79181a5580 con 0x7f7918108300 2024-03-16T09:41:11.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.596+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79181039b0 con 0x7f7918108300 2024-03-16T09:41:11.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.597+0000 7f79157fa700 1 -- 172.21.15.77:0/4205064397 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f791001b030 con 0x7f7918108300 2024-03-16T09:41:11.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.598+0000 7f79157fa700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f79000a2920 0x7f79000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:11.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.598+0000 7f7917fff700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f79000a2920 0x7f79000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:11.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.598+0000 7f79157fa700 1 -- 172.21.15.77:0/4205064397 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f79100ce640 con 0x7f7918108300 2024-03-16T09:41:11.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.598+0000 7f7917fff700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f79000a2920 0x7f79000a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7908003760 tx=0x7f7908005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:11.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.602+0000 7f79157fa700 1 -- 172.21.15.77:0/4205064397 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7910022020 con 0x7f7918108300 2024-03-16T09:41:11.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.792+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7918199c00 con 0x7f79000a2920 2024-03-16T09:41:11.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.799+0000 7f79157fa700 1 -- 172.21.15.77:0/4205064397 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258688 (secure 0 0 0) 0x7f7918199c00 con 0x7f79000a2920 2024-03-16T09:41:11.805 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f79000a2920 msgr2=0x7f79000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f79000a2920 0x7f79000a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7908003760 tx=0x7f7908005fb0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 msgr2=0x7f79181986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181986d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f791000efc0 tx=0x7f791000c3a0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 shutdown_connections 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f79000a2920 0x7f79000a4de0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7918108300 0x7f79181986d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7918102320 0x7f791819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 --2- 172.21.15.77:0/4205064397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7918102ce0 0x7f791819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 >> 172.21.15.77:0/4205064397 conn(0x7f7918079400 msgr2=0x7f7918100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 shutdown_connections 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:11.802+0000 7f791e70b700 1 -- 172.21.15.77:0/4205064397 wait complete. 2024-03-16T09:41:11.807 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T09:41:12.519 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-16T09:41:11.789788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654235,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":574,"num_read_kb":402,"num_write":338,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358380,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583444,"statfs":{"total":1151940427776,"available":1151573446656,"internally_reserved":0,"allocated":13627392,"data_stored":7484013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":44,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.861250+0000","last_change":"2024-03-16T09:39:56.184389+0000","last_active":"2024-03-16T09:40:58.861250+0000","last_peered":"2024-03-16T09:40:58.861250+0000","last_clean":"2024-03-16T09:40:58.861250+0000","last_became_active":"2024-03-16T09:39:56.184129+0000","last_became_peered":"2024-03-16T09:39:56.184129+0000","last_unstale":"2024-03-16T09:40:58.861250+0000","last_undegraded":"2024-03-16T09:40:58.861250+0000","last_fullsized":"2024-03-16T09:40:58.861250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:33.786788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865568+0000","last_change":"2024-03-16T09:39:56.193451+0000","last_active":"2024-03-16T09:40:58.865568+0000","last_peered":"2024-03-16T09:40:58.865568+0000","last_clean":"2024-03-16T09:40:58.865568+0000","last_became_active":"2024-03-16T09:39:56.187381+0000","last_became_peered":"2024-03-16T09:39:56.187381+0000","last_unstale":"2024-03-16T09:40:58.865568+0000","last_undegraded":"2024-03-16T09:40:58.865568+0000","last_fullsized":"2024-03-16T09:40:58.865568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:51.915654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865317+0000","last_change":"2024-03-16T09:39:56.183785+0000","last_active":"2024-03-16T09:40:58.865317+0000","last_peered":"2024-03-16T09:40:58.865317+0000","last_clean":"2024-03-16T09:40:58.865317+0000","last_became_active":"2024-03-16T09:39:56.183421+0000","last_became_peered":"2024-03-16T09:39:56.183421+0000","last_unstale":"2024-03-16T09:40:58.865317+0000","last_undegraded":"2024-03-16T09:40:58.865317+0000","last_fullsized":"2024-03-16T09:40:58.865317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:47.259139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863555+0000","last_change":"2024-03-16T09:39:56.195743+0000","last_active":"2024-03-16T09:40:58.863555+0000","last_peered":"2024-03-16T09:40:58.863555+0000","last_clean":"2024-03-16T09:40:58.863555+0000","last_became_active":"2024-03-16T09:39:56.194818+0000","last_became_peered":"2024-03-16T09:39:56.194818+0000","last_unstale":"2024-03-16T09:40:58.863555+0000","last_undegraded":"2024-03-16T09:40:58.863555+0000","last_fullsized":"2024-03-16T09:40:58.863555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:39:43.123879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863606+0000","last_change":"2024-03-16T09:39:56.184970+0000","last_active":"2024-03-16T09:40:58.863606+0000","last_peered":"2024-03-16T09:40:58.863606+0000","last_clean":"2024-03-16T09:40:58.863606+0000","last_became_active":"2024-03-16T09:39:56.184632+0000","last_became_peered":"2024-03-16T09:39:56.184632+0000","last_unstale":"2024-03-16T09:40:58.863606+0000","last_undegraded":"2024-03-16T09:40:58.863606+0000","last_fullsized":"2024-03-16T09:40:58.863606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:46:51.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.218928+0000","last_change":"2024-03-16T09:39:56.199688+0000","last_active":"2024-03-16T09:40:59.218928+0000","last_peered":"2024-03-16T09:40:59.218928+0000","last_clean":"2024-03-16T09:40:59.218928+0000","last_became_active":"2024-03-16T09:39:56.199094+0000","last_became_peered":"2024-03-16T09:39:56.199094+0000","last_unstale":"2024-03-16T09:40:59.218928+0000","last_undegraded":"2024-03-16T09:40:59.218928+0000","last_fullsized":"2024-03-16T09:40:59.218928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:10.855504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860444+0000","last_change":"2024-03-16T09:39:56.180816+0000","last_active":"2024-03-16T09:40:58.860444+0000","last_peered":"2024-03-16T09:40:58.860444+0000","last_clean":"2024-03-16T09:40:58.860444+0000","last_became_active":"2024-03-16T09:39:56.180499+0000","last_became_peered":"2024-03-16T09:39:56.180499+0000","last_unstale":"2024-03-16T09:40:58.860444+0000","last_undegraded":"2024-03-16T09:40:58.860444+0000","last_fullsized":"2024-03-16T09:40:58.860444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:32:09.444494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871439+0000","last_change":"2024-03-16T09:39:56.189960+0000","last_active":"2024-03-16T09:40:58.871439+0000","last_peered":"2024-03-16T09:40:58.871439+0000","last_clean":"2024-03-16T09:40:58.871439+0000","last_became_active":"2024-03-16T09:39:56.189609+0000","last_became_peered":"2024-03-16T09:39:56.189609+0000","last_unstale":"2024-03-16T09:40:58.871439+0000","last_undegraded":"2024-03-16T09:40:58.871439+0000","last_fullsized":"2024-03-16T09:40:58.871439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:50.219703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.867996+0000","last_change":"2024-03-16T09:40:58.867996+0000","last_active":"2024-03-16T09:40:58.867996+0000","last_peered":"2024-03-16T09:40:58.867996+0000","last_clean":"2024-03-16T09:40:58.867996+0000","last_became_active":"2024-03-16T09:40:58.867619+0000","last_became_peered":"2024-03-16T09:40:58.867619+0000","last_unstale":"2024-03-16T09:40:58.867996+0000","last_undegraded":"2024-03-16T09:40:58.867996+0000","last_fullsized":"2024-03-16T09:40:58.867996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:36.972983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.219160+0000","last_change":"2024-03-16T09:39:56.201584+0000","last_active":"2024-03-16T09:40:59.219160+0000","last_peered":"2024-03-16T09:40:59.219160+0000","last_clean":"2024-03-16T09:40:59.219160+0000","last_became_active":"2024-03-16T09:39:56.201320+0000","last_became_peered":"2024-03-16T09:39:56.201320+0000","last_unstale":"2024-03-16T09:40:59.219160+0000","last_undegraded":"2024-03-16T09:40:59.219160+0000","last_fullsized":"2024-03-16T09:40:59.219160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:18:00.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860869+0000","last_change":"2024-03-16T09:39:56.191125+0000","last_active":"2024-03-16T09:40:58.860869+0000","last_peered":"2024-03-16T09:40:58.860869+0000","last_clean":"2024-03-16T09:40:58.860869+0000","last_became_active":"2024-03-16T09:39:56.190795+0000","last_became_peered":"2024-03-16T09:39:56.190795+0000","last_unstale":"2024-03-16T09:40:58.860869+0000","last_undegraded":"2024-03-16T09:40:58.860869+0000","last_fullsized":"2024-03-16T09:40:58.860869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:44:10.894995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863425+0000","last_change":"2024-03-16T09:39:56.195652+0000","last_active":"2024-03-16T09:40:58.863425+0000","last_peered":"2024-03-16T09:40:58.863425+0000","last_clean":"2024-03-16T09:40:58.863425+0000","last_became_active":"2024-03-16T09:39:56.195270+0000","last_became_peered":"2024-03-16T09:39:56.195270+0000","last_unstale":"2024-03-16T09:40:58.863425+0000","last_undegraded":"2024-03-16T09:40:58.863425+0000","last_fullsized":"2024-03-16T09:40:58.863425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:03:41.546991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866140+0000","last_change":"2024-03-16T09:39:56.184622+0000","last_active":"2024-03-16T09:40:58.866140+0000","last_peered":"2024-03-16T09:40:58.866140+0000","last_clean":"2024-03-16T09:40:58.866140+0000","last_became_active":"2024-03-16T09:39:56.183532+0000","last_became_peered":"2024-03-16T09:39:56.183532+0000","last_unstale":"2024-03-16T09:40:58.866140+0000","last_undegraded":"2024-03-16T09:40:58.866140+0000","last_fullsized":"2024-03-16T09:40:58.866140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:11:24.574088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865139+0000","last_change":"2024-03-16T09:39:56.184393+0000","last_active":"2024-03-16T09:40:58.865139+0000","last_peered":"2024-03-16T09:40:58.865139+0000","last_clean":"2024-03-16T09:40:58.865139+0000","last_became_active":"2024-03-16T09:39:56.183783+0000","last_became_peered":"2024-03-16T09:39:56.183783+0000","last_unstale":"2024-03-16T09:40:58.865139+0000","last_undegraded":"2024-03-16T09:40:58.865139+0000","last_fullsized":"2024-03-16T09:40:58.865139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:16:22.131116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864979+0000","last_change":"2024-03-16T09:39:56.197365+0000","last_active":"2024-03-16T09:40:58.864979+0000","last_peered":"2024-03-16T09:40:58.864979+0000","last_clean":"2024-03-16T09:40:58.864979+0000","last_became_active":"2024-03-16T09:39:56.197020+0000","last_became_peered":"2024-03-16T09:39:56.197020+0000","last_unstale":"2024-03-16T09:40:58.864979+0000","last_undegraded":"2024-03-16T09:40:58.864979+0000","last_fullsized":"2024-03-16T09:40:58.864979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:29.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.862940+0000","last_change":"2024-03-16T09:39:56.185513+0000","last_active":"2024-03-16T09:40:58.862940+0000","last_peered":"2024-03-16T09:40:58.862940+0000","last_clean":"2024-03-16T09:40:58.862940+0000","last_became_active":"2024-03-16T09:39:56.184384+0000","last_became_peered":"2024-03-16T09:39:56.184384+0000","last_unstale":"2024-03-16T09:40:58.862940+0000","last_undegraded":"2024-03-16T09:40:58.862940+0000","last_fullsized":"2024-03-16T09:40:58.862940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:29.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.861161+0000","last_change":"2024-03-16T09:39:56.184536+0000","last_active":"2024-03-16T09:40:58.861161+0000","last_peered":"2024-03-16T09:40:58.861161+0000","last_clean":"2024-03-16T09:40:58.861161+0000","last_became_active":"2024-03-16T09:39:56.184208+0000","last_became_peered":"2024-03-16T09:39:56.184208+0000","last_unstale":"2024-03-16T09:40:58.861161+0000","last_undegraded":"2024-03-16T09:40:58.861161+0000","last_fullsized":"2024-03-16T09:40:58.861161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:14.025625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860098+0000","last_change":"2024-03-16T09:39:56.184331+0000","last_active":"2024-03-16T09:40:58.860098+0000","last_peered":"2024-03-16T09:40:58.860098+0000","last_clean":"2024-03-16T09:40:58.860098+0000","last_became_active":"2024-03-16T09:39:56.183919+0000","last_became_peered":"2024-03-16T09:39:56.183919+0000","last_unstale":"2024-03-16T09:40:58.860098+0000","last_undegraded":"2024-03-16T09:40:58.860098+0000","last_fullsized":"2024-03-16T09:40:58.860098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:59.692854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865484+0000","last_change":"2024-03-16T09:39:56.185607+0000","last_active":"2024-03-16T09:40:58.865484+0000","last_peered":"2024-03-16T09:40:58.865484+0000","last_clean":"2024-03-16T09:40:58.865484+0000","last_became_active":"2024-03-16T09:39:56.185278+0000","last_became_peered":"2024-03-16T09:39:56.185278+0000","last_unstale":"2024-03-16T09:40:58.865484+0000","last_undegraded":"2024-03-16T09:40:58.865484+0000","last_fullsized":"2024-03-16T09:40:58.865484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:20.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860105+0000","last_change":"2024-03-16T09:39:55.180852+0000","last_active":"2024-03-16T09:40:58.860105+0000","last_peered":"2024-03-16T09:40:58.860105+0000","last_clean":"2024-03-16T09:40:58.860105+0000","last_became_active":"2024-03-16T09:39:55.180496+0000","last_became_peered":"2024-03-16T09:39:55.180496+0000","last_unstale":"2024-03-16T09:40:58.860105+0000","last_undegraded":"2024-03-16T09:40:58.860105+0000","last_fullsized":"2024-03-16T09:40:58.860105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:15:28.548458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871431+0000","last_change":"2024-03-16T09:39:56.190110+0000","last_active":"2024-03-16T09:40:58.871431+0000","last_peered":"2024-03-16T09:40:58.871431+0000","last_clean":"2024-03-16T09:40:58.871431+0000","last_became_active":"2024-03-16T09:39:56.189711+0000","last_became_peered":"2024-03-16T09:39:56.189711+0000","last_unstale":"2024-03-16T09:40:58.871431+0000","last_undegraded":"2024-03-16T09:40:58.871431+0000","last_fullsized":"2024-03-16T09:40:58.871431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:17.353457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871621+0000","last_change":"2024-03-16T09:39:55.171142+0000","last_active":"2024-03-16T09:40:58.871621+0000","last_peered":"2024-03-16T09:40:58.871621+0000","last_clean":"2024-03-16T09:40:58.871621+0000","last_became_active":"2024-03-16T09:39:55.170793+0000","last_became_peered":"2024-03-16T09:39:55.170793+0000","last_unstale":"2024-03-16T09:40:58.871621+0000","last_undegraded":"2024-03-16T09:40:58.871621+0000","last_fullsized":"2024-03-16T09:40:58.871621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:36.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865733+0000","last_change":"2024-03-16T09:39:56.186802+0000","last_active":"2024-03-16T09:40:58.865733+0000","last_peered":"2024-03-16T09:40:58.865733+0000","last_clean":"2024-03-16T09:40:58.865733+0000","last_became_active":"2024-03-16T09:39:56.186398+0000","last_became_peered":"2024-03-16T09:39:56.186398+0000","last_unstale":"2024-03-16T09:40:58.865733+0000","last_undegraded":"2024-03-16T09:40:58.865733+0000","last_fullsized":"2024-03-16T09:40:58.865733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:05:56.209864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864171+0000","last_change":"2024-03-16T09:39:55.181894+0000","last_active":"2024-03-16T09:40:58.864171+0000","last_peered":"2024-03-16T09:40:58.864171+0000","last_clean":"2024-03-16T09:40:58.864171+0000","last_became_active":"2024-03-16T09:39:55.179434+0000","last_became_peered":"2024-03-16T09:39:55.179434+0000","last_unstale":"2024-03-16T09:40:58.864171+0000","last_undegraded":"2024-03-16T09:40:58.864171+0000","last_fullsized":"2024-03-16T09:40:58.864171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:16.153928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.862941+0000","last_change":"2024-03-16T09:39:56.188464+0000","last_active":"2024-03-16T09:40:58.862941+0000","last_peered":"2024-03-16T09:40:58.862941+0000","last_clean":"2024-03-16T09:40:58.862941+0000","last_became_active":"2024-03-16T09:39:56.188125+0000","last_became_peered":"2024-03-16T09:39:56.188125+0000","last_unstale":"2024-03-16T09:40:58.862941+0000","last_undegraded":"2024-03-16T09:40:58.862941+0000","last_fullsized":"2024-03-16T09:40:58.862941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:46:49.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863856+0000","last_change":"2024-03-16T09:39:55.179379+0000","last_active":"2024-03-16T09:40:58.863856+0000","last_peered":"2024-03-16T09:40:58.863856+0000","last_clean":"2024-03-16T09:40:58.863856+0000","last_became_active":"2024-03-16T09:39:55.178755+0000","last_became_peered":"2024-03-16T09:39:55.178755+0000","last_unstale":"2024-03-16T09:40:58.863856+0000","last_undegraded":"2024-03-16T09:40:58.863856+0000","last_fullsized":"2024-03-16T09:40:58.863856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:07.131417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863602+0000","last_change":"2024-03-16T09:39:56.196655+0000","last_active":"2024-03-16T09:40:58.863602+0000","last_peered":"2024-03-16T09:40:58.863602+0000","last_clean":"2024-03-16T09:40:58.863602+0000","last_became_active":"2024-03-16T09:39:56.196344+0000","last_became_peered":"2024-03-16T09:39:56.196344+0000","last_unstale":"2024-03-16T09:40:58.863602+0000","last_undegraded":"2024-03-16T09:40:58.863602+0000","last_fullsized":"2024-03-16T09:40:58.863602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:12:03.920319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:05.804950+0000","last_change":"2024-03-16T09:39:55.182012+0000","last_active":"2024-03-16T09:41:05.804950+0000","last_peered":"2024-03-16T09:41:05.804950+0000","last_clean":"2024-03-16T09:41:05.804950+0000","last_became_active":"2024-03-16T09:39:55.178651+0000","last_became_peered":"2024-03-16T09:39:55.178651+0000","last_unstale":"2024-03-16T09:41:05.804950+0000","last_undegraded":"2024-03-16T09:41:05.804950+0000","last_fullsized":"2024-03-16T09:41:05.804950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:50:19.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863596+0000","last_change":"2024-03-16T09:39:56.179645+0000","last_active":"2024-03-16T09:40:58.863596+0000","last_peered":"2024-03-16T09:40:58.863596+0000","last_clean":"2024-03-16T09:40:58.863596+0000","last_became_active":"2024-03-16T09:39:56.179257+0000","last_became_peered":"2024-03-16T09:39:56.179257+0000","last_unstale":"2024-03-16T09:40:58.863596+0000","last_undegraded":"2024-03-16T09:40:58.863596+0000","last_fullsized":"2024-03-16T09:40:58.863596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:30:09.104630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.218444+0000","last_change":"2024-03-16T09:39:55.178257+0000","last_active":"2024-03-16T09:40:59.218444+0000","last_peered":"2024-03-16T09:40:59.218444+0000","last_clean":"2024-03-16T09:40:59.218444+0000","last_became_active":"2024-03-16T09:39:55.177984+0000","last_became_peered":"2024-03-16T09:39:55.177984+0000","last_unstale":"2024-03-16T09:40:59.218444+0000","last_undegraded":"2024-03-16T09:40:59.218444+0000","last_fullsized":"2024-03-16T09:40:59.218444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.385624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863340+0000","last_change":"2024-03-16T09:39:56.196821+0000","last_active":"2024-03-16T09:40:58.863340+0000","last_peered":"2024-03-16T09:40:58.863340+0000","last_clean":"2024-03-16T09:40:58.863340+0000","last_became_active":"2024-03-16T09:39:56.196522+0000","last_became_peered":"2024-03-16T09:39:56.196522+0000","last_unstale":"2024-03-16T09:40:58.863340+0000","last_undegraded":"2024-03-16T09:40:58.863340+0000","last_fullsized":"2024-03-16T09:40:58.863340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:31.103530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865998+0000","last_change":"2024-03-16T09:39:55.170290+0000","last_active":"2024-03-16T09:40:58.865998+0000","last_peered":"2024-03-16T09:40:58.865998+0000","last_clean":"2024-03-16T09:40:58.865998+0000","last_became_active":"2024-03-16T09:39:55.169956+0000","last_became_peered":"2024-03-16T09:39:55.169956+0000","last_unstale":"2024-03-16T09:40:58.865998+0000","last_undegraded":"2024-03-16T09:40:58.865998+0000","last_fullsized":"2024-03-16T09:40:58.865998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:01:31.606917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871481+0000","last_change":"2024-03-16T09:39:56.189155+0000","last_active":"2024-03-16T09:40:58.871481+0000","last_peered":"2024-03-16T09:40:58.871481+0000","last_clean":"2024-03-16T09:40:58.871481+0000","last_became_active":"2024-03-16T09:39:56.188827+0000","last_became_peered":"2024-03-16T09:39:56.188827+0000","last_unstale":"2024-03-16T09:40:58.871481+0000","last_undegraded":"2024-03-16T09:40:58.871481+0000","last_fullsized":"2024-03-16T09:40:58.871481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:54.370754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863093+0000","last_change":"2024-03-16T09:39:55.181886+0000","last_active":"2024-03-16T09:40:58.863093+0000","last_peered":"2024-03-16T09:40:58.863093+0000","last_clean":"2024-03-16T09:40:58.863093+0000","last_became_active":"2024-03-16T09:39:55.179836+0000","last_became_peered":"2024-03-16T09:39:55.179836+0000","last_unstale":"2024-03-16T09:40:58.863093+0000","last_undegraded":"2024-03-16T09:40:58.863093+0000","last_fullsized":"2024-03-16T09:40:58.863093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:52:38.772861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.859838+0000","last_change":"2024-03-16T09:39:56.185892+0000","last_active":"2024-03-16T09:40:58.859838+0000","last_peered":"2024-03-16T09:40:58.859838+0000","last_clean":"2024-03-16T09:40:58.859838+0000","last_became_active":"2024-03-16T09:39:56.185459+0000","last_became_peered":"2024-03-16T09:39:56.185459+0000","last_unstale":"2024-03-16T09:40:58.859838+0000","last_undegraded":"2024-03-16T09:40:58.859838+0000","last_fullsized":"2024-03-16T09:40:58.859838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:10.421054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:08.807330+0000","last_change":"2024-03-16T09:39:55.177607+0000","last_active":"2024-03-16T09:41:08.807330+0000","last_peered":"2024-03-16T09:41:08.807330+0000","last_clean":"2024-03-16T09:41:08.807330+0000","last_became_active":"2024-03-16T09:39:55.177265+0000","last_became_peered":"2024-03-16T09:39:55.177265+0000","last_unstale":"2024-03-16T09:41:08.807330+0000","last_undegraded":"2024-03-16T09:41:08.807330+0000","last_fullsized":"2024-03-16T09:41:08.807330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:47.849041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863281+0000","last_change":"2024-03-16T09:39:56.189172+0000","last_active":"2024-03-16T09:40:58.863281+0000","last_peered":"2024-03-16T09:40:58.863281+0000","last_clean":"2024-03-16T09:40:58.863281+0000","last_became_active":"2024-03-16T09:39:56.188798+0000","last_became_peered":"2024-03-16T09:39:56.188798+0000","last_unstale":"2024-03-16T09:40:58.863281+0000","last_undegraded":"2024-03-16T09:40:58.863281+0000","last_fullsized":"2024-03-16T09:40:58.863281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:02:11.119293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860780+0000","last_change":"2024-03-16T09:39:55.180369+0000","last_active":"2024-03-16T09:40:58.860780+0000","last_peered":"2024-03-16T09:40:58.860780+0000","last_clean":"2024-03-16T09:40:58.860780+0000","last_became_active":"2024-03-16T09:39:55.180053+0000","last_became_peered":"2024-03-16T09:39:55.180053+0000","last_unstale":"2024-03-16T09:40:58.860780+0000","last_undegraded":"2024-03-16T09:40:58.860780+0000","last_fullsized":"2024-03-16T09:40:58.860780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:44.990028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865703+0000","last_change":"2024-03-16T09:39:56.198454+0000","last_active":"2024-03-16T09:40:58.865703+0000","last_peered":"2024-03-16T09:40:58.865703+0000","last_clean":"2024-03-16T09:40:58.865703+0000","last_became_active":"2024-03-16T09:39:56.198097+0000","last_became_peered":"2024-03-16T09:39:56.198097+0000","last_unstale":"2024-03-16T09:40:58.865703+0000","last_undegraded":"2024-03-16T09:40:58.865703+0000","last_fullsized":"2024-03-16T09:40:58.865703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:12.494363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863700+0000","last_change":"2024-03-16T09:39:55.174221+0000","last_active":"2024-03-16T09:40:58.863700+0000","last_peered":"2024-03-16T09:40:58.863700+0000","last_clean":"2024-03-16T09:40:58.863700+0000","last_became_active":"2024-03-16T09:39:55.173920+0000","last_became_peered":"2024-03-16T09:39:55.173920+0000","last_unstale":"2024-03-16T09:40:58.863700+0000","last_undegraded":"2024-03-16T09:40:58.863700+0000","last_fullsized":"2024-03-16T09:40:58.863700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:25.108414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863385+0000","last_change":"2024-03-16T09:39:56.196849+0000","last_active":"2024-03-16T09:40:58.863385+0000","last_peered":"2024-03-16T09:40:58.863385+0000","last_clean":"2024-03-16T09:40:58.863385+0000","last_became_active":"2024-03-16T09:39:56.196515+0000","last_became_peered":"2024-03-16T09:39:56.196515+0000","last_unstale":"2024-03-16T09:40:58.863385+0000","last_undegraded":"2024-03-16T09:40:58.863385+0000","last_fullsized":"2024-03-16T09:40:58.863385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:03.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'29","reported_seq":198,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:09.808047+0000","last_change":"2024-03-16T09:39:55.177788+0000","last_active":"2024-03-16T09:41:09.808047+0000","last_peered":"2024-03-16T09:41:09.808047+0000","last_clean":"2024-03-16T09:41:09.808047+0000","last_became_active":"2024-03-16T09:39:55.177443+0000","last_became_peered":"2024-03-16T09:39:55.177443+0000","last_unstale":"2024-03-16T09:41:09.808047+0000","last_undegraded":"2024-03-16T09:41:09.808047+0000","last_fullsized":"2024-03-16T09:41:09.808047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:39.221712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":150,"num_read_kb":133,"num_write":30,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863958+0000","last_change":"2024-03-16T09:39:56.178976+0000","last_active":"2024-03-16T09:40:58.863958+0000","last_peered":"2024-03-16T09:40:58.863958+0000","last_clean":"2024-03-16T09:40:58.863958+0000","last_became_active":"2024-03-16T09:39:56.178455+0000","last_became_peered":"2024-03-16T09:39:56.178455+0000","last_unstale":"2024-03-16T09:40:58.863958+0000","last_undegraded":"2024-03-16T09:40:58.863958+0000","last_fullsized":"2024-03-16T09:40:58.863958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:52:08.064792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865191+0000","last_change":"2024-03-16T09:39:55.182579+0000","last_active":"2024-03-16T09:40:58.865191+0000","last_peered":"2024-03-16T09:40:58.865191+0000","last_clean":"2024-03-16T09:40:58.865191+0000","last_became_active":"2024-03-16T09:39:55.181298+0000","last_became_peered":"2024-03-16T09:39:55.181298+0000","last_unstale":"2024-03-16T09:40:58.865191+0000","last_undegraded":"2024-03-16T09:40:58.865191+0000","last_fullsized":"2024-03-16T09:40:58.865191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:35.768812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866091+0000","last_change":"2024-03-16T09:39:56.188519+0000","last_active":"2024-03-16T09:40:58.866091+0000","last_peered":"2024-03-16T09:40:58.866091+0000","last_clean":"2024-03-16T09:40:58.866091+0000","last_became_active":"2024-03-16T09:39:56.188031+0000","last_became_peered":"2024-03-16T09:39:56.188031+0000","last_unstale":"2024-03-16T09:40:58.866091+0000","last_undegraded":"2024-03-16T09:40:58.866091+0000","last_fullsized":"2024-03-16T09:40:58.866091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:20:31.765657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864304+0000","last_change":"2024-03-16T09:39:55.177523+0000","last_active":"2024-03-16T09:40:58.864304+0000","last_peered":"2024-03-16T09:40:58.864304+0000","last_clean":"2024-03-16T09:40:58.864304+0000","last_became_active":"2024-03-16T09:39:55.177143+0000","last_became_peered":"2024-03-16T09:39:55.177143+0000","last_unstale":"2024-03-16T09:40:58.864304+0000","last_undegraded":"2024-03-16T09:40:58.864304+0000","last_fullsized":"2024-03-16T09:40:58.864304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:05.357611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.867151+0000","last_change":"2024-03-16T09:40:58.867151+0000","last_active":"2024-03-16T09:40:58.867151+0000","last_peered":"2024-03-16T09:40:58.867151+0000","last_clean":"2024-03-16T09:40:58.867151+0000","last_became_active":"2024-03-16T09:40:58.866756+0000","last_became_peered":"2024-03-16T09:40:58.866756+0000","last_unstale":"2024-03-16T09:40:58.867151+0000","last_undegraded":"2024-03-16T09:40:58.867151+0000","last_fullsized":"2024-03-16T09:40:58.867151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:53.323964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:07.818161+0000","last_change":"2024-03-16T09:39:55.175419+0000","last_active":"2024-03-16T09:41:07.818161+0000","last_peered":"2024-03-16T09:41:07.818161+0000","last_clean":"2024-03-16T09:41:07.818161+0000","last_became_active":"2024-03-16T09:39:55.175126+0000","last_became_peered":"2024-03-16T09:39:55.175126+0000","last_unstale":"2024-03-16T09:41:07.818161+0000","last_undegraded":"2024-03-16T09:41:07.818161+0000","last_fullsized":"2024-03-16T09:41:07.818161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:11.696911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860379+0000","last_change":"2024-03-16T09:39:56.176714+0000","last_active":"2024-03-16T09:40:58.860379+0000","last_peered":"2024-03-16T09:40:58.860379+0000","last_clean":"2024-03-16T09:40:58.860379+0000","last_became_active":"2024-03-16T09:39:56.176245+0000","last_became_peered":"2024-03-16T09:39:56.176245+0000","last_unstale":"2024-03-16T09:40:58.860379+0000","last_undegraded":"2024-03-16T09:40:58.860379+0000","last_fullsized":"2024-03-16T09:40:58.860379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:32.430207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863725+0000","last_change":"2024-03-16T09:39:55.181486+0000","last_active":"2024-03-16T09:40:58.863725+0000","last_peered":"2024-03-16T09:40:58.863725+0000","last_clean":"2024-03-16T09:40:58.863725+0000","last_became_active":"2024-03-16T09:39:55.180128+0000","last_became_peered":"2024-03-16T09:39:55.180128+0000","last_unstale":"2024-03-16T09:40:58.863725+0000","last_undegraded":"2024-03-16T09:40:58.863725+0000","last_fullsized":"2024-03-16T09:40:58.863725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:45.323870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864169+0000","last_change":"2024-03-16T09:39:56.196692+0000","last_active":"2024-03-16T09:40:58.864169+0000","last_peered":"2024-03-16T09:40:58.864169+0000","last_clean":"2024-03-16T09:40:58.864169+0000","last_became_active":"2024-03-16T09:39:56.196403+0000","last_became_peered":"2024-03-16T09:39:56.196403+0000","last_unstale":"2024-03-16T09:40:58.864169+0000","last_undegraded":"2024-03-16T09:40:58.864169+0000","last_fullsized":"2024-03-16T09:40:58.864169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:00:29.426476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871091+0000","last_change":"2024-03-16T09:40:58.870587+0000","last_active":"2024-03-16T09:40:58.871091+0000","last_peered":"2024-03-16T09:40:58.871091+0000","last_clean":"2024-03-16T09:40:58.871091+0000","last_became_active":"2024-03-16T09:40:58.864977+0000","last_became_peered":"2024-03-16T09:40:58.864977+0000","last_unstale":"2024-03-16T09:40:58.871091+0000","last_undegraded":"2024-03-16T09:40:58.871091+0000","last_fullsized":"2024-03-16T09:40:58.871091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:18.189307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866074+0000","last_change":"2024-03-16T09:40:58.866074+0000","last_active":"2024-03-16T09:40:58.866074+0000","last_peered":"2024-03-16T09:40:58.866074+0000","last_clean":"2024-03-16T09:40:58.866074+0000","last_became_active":"2024-03-16T09:40:58.865711+0000","last_became_peered":"2024-03-16T09:40:58.865711+0000","last_unstale":"2024-03-16T09:40:58.866074+0000","last_undegraded":"2024-03-16T09:40:58.866074+0000","last_fullsized":"2024-03-16T09:40:58.866074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:57.137802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.859988+0000","last_change":"2024-03-16T09:39:55.175666+0000","last_active":"2024-03-16T09:40:58.859988+0000","last_peered":"2024-03-16T09:40:58.859988+0000","last_clean":"2024-03-16T09:40:58.859988+0000","last_became_active":"2024-03-16T09:39:55.175337+0000","last_became_peered":"2024-03-16T09:39:55.175337+0000","last_unstale":"2024-03-16T09:40:58.859988+0000","last_undegraded":"2024-03-16T09:40:58.859988+0000","last_fullsized":"2024-03-16T09:40:58.859988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:45:13.935669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.219039+0000","last_change":"2024-03-16T09:39:56.184878+0000","last_active":"2024-03-16T09:40:59.219039+0000","last_peered":"2024-03-16T09:40:59.219039+0000","last_clean":"2024-03-16T09:40:59.219039+0000","last_became_active":"2024-03-16T09:39:56.184480+0000","last_became_peered":"2024-03-16T09:39:56.184480+0000","last_unstale":"2024-03-16T09:40:59.219039+0000","last_undegraded":"2024-03-16T09:40:59.219039+0000","last_fullsized":"2024-03-16T09:40:59.219039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:40:01.286993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860584+0000","last_change":"2024-03-16T09:39:55.179189+0000","last_active":"2024-03-16T09:40:58.860584+0000","last_peered":"2024-03-16T09:40:58.860584+0000","last_clean":"2024-03-16T09:40:58.860584+0000","last_became_active":"2024-03-16T09:39:55.178859+0000","last_became_peered":"2024-03-16T09:39:55.178859+0000","last_unstale":"2024-03-16T09:40:58.860584+0000","last_undegraded":"2024-03-16T09:40:58.860584+0000","last_fullsized":"2024-03-16T09:40:58.860584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:54:15.498386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.219025+0000","last_change":"2024-03-16T09:39:56.200665+0000","last_active":"2024-03-16T09:40:59.219025+0000","last_peered":"2024-03-16T09:40:59.219025+0000","last_clean":"2024-03-16T09:40:59.219025+0000","last_became_active":"2024-03-16T09:39:56.200326+0000","last_became_peered":"2024-03-16T09:39:56.200326+0000","last_unstale":"2024-03-16T09:40:59.219025+0000","last_undegraded":"2024-03-16T09:40:59.219025+0000","last_fullsized":"2024-03-16T09:40:59.219025+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:34.778630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863128+0000","last_change":"2024-03-16T09:39:56.182670+0000","last_active":"2024-03-16T09:40:58.863128+0000","last_peered":"2024-03-16T09:40:58.863128+0000","last_clean":"2024-03-16T09:40:58.863128+0000","last_became_active":"2024-03-16T09:39:56.182009+0000","last_became_peered":"2024-03-16T09:39:56.182009+0000","last_unstale":"2024-03-16T09:40:58.863128+0000","last_undegraded":"2024-03-16T09:40:58.863128+0000","last_fullsized":"2024-03-16T09:40:58.863128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:28:03.106071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:09.819946+0000","last_change":"2024-03-16T09:39:55.171609+0000","last_active":"2024-03-16T09:41:09.819946+0000","last_peered":"2024-03-16T09:41:09.819946+0000","last_clean":"2024-03-16T09:41:09.819946+0000","last_became_active":"2024-03-16T09:39:55.171301+0000","last_became_peered":"2024-03-16T09:39:55.171301+0000","last_unstale":"2024-03-16T09:41:09.819946+0000","last_undegraded":"2024-03-16T09:41:09.819946+0000","last_fullsized":"2024-03-16T09:41:09.819946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:28.265342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.867266+0000","last_change":"2024-03-16T09:40:58.865311+0000","last_active":"2024-03-16T09:40:58.867266+0000","last_peered":"2024-03-16T09:40:58.867266+0000","last_clean":"2024-03-16T09:40:58.867266+0000","last_became_active":"2024-03-16T09:40:58.864918+0000","last_became_peered":"2024-03-16T09:40:58.864918+0000","last_unstale":"2024-03-16T09:40:58.867266+0000","last_undegraded":"2024-03-16T09:40:58.867266+0000","last_fullsized":"2024-03-16T09:40:58.867266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863236+0000","last_change":"2024-03-16T09:39:55.172824+0000","last_active":"2024-03-16T09:40:58.863236+0000","last_peered":"2024-03-16T09:40:58.863236+0000","last_clean":"2024-03-16T09:40:58.863236+0000","last_became_active":"2024-03-16T09:39:55.172493+0000","last_became_peered":"2024-03-16T09:39:55.172493+0000","last_unstale":"2024-03-16T09:40:58.863236+0000","last_undegraded":"2024-03-16T09:40:58.863236+0000","last_fullsized":"2024-03-16T09:40:58.863236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:39.188176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860453+0000","last_change":"2024-03-16T09:38:43.693644+0000","last_active":"2024-03-16T09:40:58.860453+0000","last_peered":"2024-03-16T09:40:58.860453+0000","last_clean":"2024-03-16T09:40:58.860453+0000","last_became_active":"2024-03-16T09:38:43.693202+0000","last_became_peered":"2024-03-16T09:38:43.693202+0000","last_unstale":"2024-03-16T09:40:58.860453+0000","last_undegraded":"2024-03-16T09:40:58.860453+0000","last_fullsized":"2024-03-16T09:40:58.860453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_clean_scrub_stamp":"2024-03-16T09:35:24.153508+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:43:06.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865878+0000","last_change":"2024-03-16T09:39:56.197109+0000","last_active":"2024-03-16T09:40:58.865878+0000","last_peered":"2024-03-16T09:40:58.865878+0000","last_clean":"2024-03-16T09:40:58.865878+0000","last_became_active":"2024-03-16T09:39:56.196775+0000","last_became_peered":"2024-03-16T09:39:56.196775+0000","last_unstale":"2024-03-16T09:40:58.865878+0000","last_undegraded":"2024-03-16T09:40:58.865878+0000","last_fullsized":"2024-03-16T09:40:58.865878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:29:10.401423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860552+0000","last_change":"2024-03-16T09:39:55.175744+0000","last_active":"2024-03-16T09:40:58.860552+0000","last_peered":"2024-03-16T09:40:58.860552+0000","last_clean":"2024-03-16T09:40:58.860552+0000","last_became_active":"2024-03-16T09:39:55.175445+0000","last_became_peered":"2024-03-16T09:39:55.175445+0000","last_unstale":"2024-03-16T09:40:58.860552+0000","last_undegraded":"2024-03-16T09:40:58.860552+0000","last_fullsized":"2024-03-16T09:40:58.860552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:39.494340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864068+0000","last_change":"2024-03-16T09:39:55.183441+0000","last_active":"2024-03-16T09:40:58.864068+0000","last_peered":"2024-03-16T09:40:58.864068+0000","last_clean":"2024-03-16T09:40:58.864068+0000","last_became_active":"2024-03-16T09:39:55.183115+0000","last_became_peered":"2024-03-16T09:39:55.183115+0000","last_unstale":"2024-03-16T09:40:58.864068+0000","last_undegraded":"2024-03-16T09:40:58.864068+0000","last_fullsized":"2024-03-16T09:40:58.864068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:55:31.525201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866116+0000","last_change":"2024-03-16T09:39:56.179205+0000","last_active":"2024-03-16T09:40:58.866116+0000","last_peered":"2024-03-16T09:40:58.866116+0000","last_clean":"2024-03-16T09:40:58.866116+0000","last_became_active":"2024-03-16T09:39:56.178849+0000","last_became_peered":"2024-03-16T09:39:56.178849+0000","last_unstale":"2024-03-16T09:40:58.866116+0000","last_undegraded":"2024-03-16T09:40:58.866116+0000","last_fullsized":"2024-03-16T09:40:58.866116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:10.016099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860157+0000","last_change":"2024-03-16T09:39:55.180859+0000","last_active":"2024-03-16T09:40:58.860157+0000","last_peered":"2024-03-16T09:40:58.860157+0000","last_clean":"2024-03-16T09:40:58.860157+0000","last_became_active":"2024-03-16T09:39:55.180514+0000","last_became_peered":"2024-03-16T09:39:55.180514+0000","last_unstale":"2024-03-16T09:40:58.860157+0000","last_undegraded":"2024-03-16T09:40:58.860157+0000","last_fullsized":"2024-03-16T09:40:58.860157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:46.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871538+0000","last_change":"2024-03-16T09:39:56.187626+0000","last_active":"2024-03-16T09:40:58.871538+0000","last_peered":"2024-03-16T09:40:58.871538+0000","last_clean":"2024-03-16T09:40:58.871538+0000","last_became_active":"2024-03-16T09:39:56.187284+0000","last_became_peered":"2024-03-16T09:39:56.187284+0000","last_unstale":"2024-03-16T09:40:58.871538+0000","last_undegraded":"2024-03-16T09:40:58.871538+0000","last_fullsized":"2024-03-16T09:40:58.871538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:23.511603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864099+0000","last_change":"2024-03-16T09:39:55.172912+0000","last_active":"2024-03-16T09:40:58.864099+0000","last_peered":"2024-03-16T09:40:58.864099+0000","last_clean":"2024-03-16T09:40:58.864099+0000","last_became_active":"2024-03-16T09:39:55.172592+0000","last_became_peered":"2024-03-16T09:39:55.172592+0000","last_unstale":"2024-03-16T09:40:58.864099+0000","last_undegraded":"2024-03-16T09:40:58.864099+0000","last_fullsized":"2024-03-16T09:40:58.864099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:24.344588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865663+0000","last_change":"2024-03-16T09:39:56.197652+0000","last_active":"2024-03-16T09:40:58.865663+0000","last_peered":"2024-03-16T09:40:58.865663+0000","last_clean":"2024-03-16T09:40:58.865663+0000","last_became_active":"2024-03-16T09:39:56.197230+0000","last_became_peered":"2024-03-16T09:39:56.197230+0000","last_unstale":"2024-03-16T09:40:58.865663+0000","last_undegraded":"2024-03-16T09:40:58.865663+0000","last_fullsized":"2024-03-16T09:40:58.865663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:45:08.661372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.869780+0000","last_change":"2024-03-16T09:40:58.869780+0000","last_active":"2024-03-16T09:40:58.869780+0000","last_peered":"2024-03-16T09:40:58.869780+0000","last_clean":"2024-03-16T09:40:58.869780+0000","last_became_active":"2024-03-16T09:40:58.866452+0000","last_became_peered":"2024-03-16T09:40:58.866452+0000","last_unstale":"2024-03-16T09:40:58.869780+0000","last_undegraded":"2024-03-16T09:40:58.869780+0000","last_fullsized":"2024-03-16T09:40:58.869780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:07:21.751383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863491+0000","last_change":"2024-03-16T09:39:56.179326+0000","last_active":"2024-03-16T09:40:58.863491+0000","last_peered":"2024-03-16T09:40:58.863491+0000","last_clean":"2024-03-16T09:40:58.863491+0000","last_became_active":"2024-03-16T09:39:56.178681+0000","last_became_peered":"2024-03-16T09:39:56.178681+0000","last_unstale":"2024-03-16T09:40:58.863491+0000","last_undegraded":"2024-03-16T09:40:58.863491+0000","last_fullsized":"2024-03-16T09:40:58.863491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:26.137807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871338+0000","last_change":"2024-03-16T09:39:55.173820+0000","last_active":"2024-03-16T09:40:58.871338+0000","last_peered":"2024-03-16T09:40:58.871338+0000","last_clean":"2024-03-16T09:40:58.871338+0000","last_became_active":"2024-03-16T09:39:55.173472+0000","last_became_peered":"2024-03-16T09:39:55.173472+0000","last_unstale":"2024-03-16T09:40:58.871338+0000","last_undegraded":"2024-03-16T09:40:58.871338+0000","last_fullsized":"2024-03-16T09:40:58.871338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:14.082367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860822+0000","last_change":"2024-03-16T09:39:56.178535+0000","last_active":"2024-03-16T09:40:58.860822+0000","last_peered":"2024-03-16T09:40:58.860822+0000","last_clean":"2024-03-16T09:40:58.860822+0000","last_became_active":"2024-03-16T09:39:56.178210+0000","last_became_peered":"2024-03-16T09:39:56.178210+0000","last_unstale":"2024-03-16T09:40:58.860822+0000","last_undegraded":"2024-03-16T09:40:58.860822+0000","last_fullsized":"2024-03-16T09:40:58.860822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:18.680469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864920+0000","last_change":"2024-03-16T09:39:55.183074+0000","last_active":"2024-03-16T09:40:58.864920+0000","last_peered":"2024-03-16T09:40:58.864920+0000","last_clean":"2024-03-16T09:40:58.864920+0000","last_became_active":"2024-03-16T09:39:55.182695+0000","last_became_peered":"2024-03-16T09:39:55.182695+0000","last_unstale":"2024-03-16T09:40:58.864920+0000","last_undegraded":"2024-03-16T09:40:58.864920+0000","last_fullsized":"2024-03-16T09:40:58.864920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:51.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871544+0000","last_change":"2024-03-16T09:39:56.193434+0000","last_active":"2024-03-16T09:40:58.871544+0000","last_peered":"2024-03-16T09:40:58.871544+0000","last_clean":"2024-03-16T09:40:58.871544+0000","last_became_active":"2024-03-16T09:39:56.193051+0000","last_became_peered":"2024-03-16T09:39:56.193051+0000","last_unstale":"2024-03-16T09:40:58.871544+0000","last_undegraded":"2024-03-16T09:40:58.871544+0000","last_fullsized":"2024-03-16T09:40:58.871544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:12:26.767117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863710+0000","last_change":"2024-03-16T09:39:55.179916+0000","last_active":"2024-03-16T09:40:58.863710+0000","last_peered":"2024-03-16T09:40:58.863710+0000","last_clean":"2024-03-16T09:40:58.863710+0000","last_became_active":"2024-03-16T09:39:55.179635+0000","last_became_peered":"2024-03-16T09:39:55.179635+0000","last_unstale":"2024-03-16T09:40:58.863710+0000","last_undegraded":"2024-03-16T09:40:58.863710+0000","last_fullsized":"2024-03-16T09:40:58.863710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:08.955039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860659+0000","last_change":"2024-03-16T09:39:56.194044+0000","last_active":"2024-03-16T09:40:58.860659+0000","last_peered":"2024-03-16T09:40:58.860659+0000","last_clean":"2024-03-16T09:40:58.860659+0000","last_became_active":"2024-03-16T09:39:56.193490+0000","last_became_peered":"2024-03-16T09:39:56.193490+0000","last_unstale":"2024-03-16T09:40:58.860659+0000","last_undegraded":"2024-03-16T09:40:58.860659+0000","last_fullsized":"2024-03-16T09:40:58.860659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:19:59.916892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865495+0000","last_change":"2024-03-16T09:39:55.182652+0000","last_active":"2024-03-16T09:40:58.865495+0000","last_peered":"2024-03-16T09:40:58.865495+0000","last_clean":"2024-03-16T09:40:58.865495+0000","last_became_active":"2024-03-16T09:39:55.181432+0000","last_became_peered":"2024-03-16T09:39:55.181432+0000","last_unstale":"2024-03-16T09:40:58.865495+0000","last_undegraded":"2024-03-16T09:40:58.865495+0000","last_fullsized":"2024-03-16T09:40:58.865495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:34.959845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860880+0000","last_change":"2024-03-16T09:39:56.188970+0000","last_active":"2024-03-16T09:40:58.860880+0000","last_peered":"2024-03-16T09:40:58.860880+0000","last_clean":"2024-03-16T09:40:58.860880+0000","last_became_active":"2024-03-16T09:39:56.188607+0000","last_became_peered":"2024-03-16T09:39:56.188607+0000","last_unstale":"2024-03-16T09:40:58.860880+0000","last_undegraded":"2024-03-16T09:40:58.860880+0000","last_fullsized":"2024-03-16T09:40:58.860880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:39:24.532114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863497+0000","last_change":"2024-03-16T09:39:55.181908+0000","last_active":"2024-03-16T09:40:58.863497+0000","last_peered":"2024-03-16T09:40:58.863497+0000","last_clean":"2024-03-16T09:40:58.863497+0000","last_became_active":"2024-03-16T09:39:55.181055+0000","last_became_peered":"2024-03-16T09:39:55.181055+0000","last_unstale":"2024-03-16T09:40:58.863497+0000","last_undegraded":"2024-03-16T09:40:58.863497+0000","last_fullsized":"2024-03-16T09:40:58.863497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:38.256664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860744+0000","last_change":"2024-03-16T09:39:56.189743+0000","last_active":"2024-03-16T09:40:58.860744+0000","last_peered":"2024-03-16T09:40:58.860744+0000","last_clean":"2024-03-16T09:40:58.860744+0000","last_became_active":"2024-03-16T09:39:56.189339+0000","last_became_peered":"2024-03-16T09:39:56.189339+0000","last_unstale":"2024-03-16T09:40:58.860744+0000","last_undegraded":"2024-03-16T09:40:58.860744+0000","last_fullsized":"2024-03-16T09:40:58.860744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:54.694191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.874651+0000","last_change":"2024-03-16T09:40:58.874651+0000","last_active":"2024-03-16T09:40:58.874651+0000","last_peered":"2024-03-16T09:40:58.874651+0000","last_clean":"2024-03-16T09:40:58.874651+0000","last_became_active":"2024-03-16T09:40:58.872329+0000","last_became_peered":"2024-03-16T09:40:58.872329+0000","last_unstale":"2024-03-16T09:40:58.874651+0000","last_undegraded":"2024-03-16T09:40:58.874651+0000","last_fullsized":"2024-03-16T09:40:58.874651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:05.424197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860934+0000","last_change":"2024-03-16T09:39:56.181636+0000","last_active":"2024-03-16T09:40:58.860934+0000","last_peered":"2024-03-16T09:40:58.860934+0000","last_clean":"2024-03-16T09:40:58.860934+0000","last_became_active":"2024-03-16T09:39:56.181309+0000","last_became_peered":"2024-03-16T09:39:56.181309+0000","last_unstale":"2024-03-16T09:40:58.860934+0000","last_undegraded":"2024-03-16T09:40:58.860934+0000","last_fullsized":"2024-03-16T09:40:58.860934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:36.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860897+0000","last_change":"2024-03-16T09:39:56.186640+0000","last_active":"2024-03-16T09:40:58.860897+0000","last_peered":"2024-03-16T09:40:58.860897+0000","last_clean":"2024-03-16T09:40:58.860897+0000","last_became_active":"2024-03-16T09:39:56.186311+0000","last_became_peered":"2024-03-16T09:39:56.186311+0000","last_unstale":"2024-03-16T09:40:58.860897+0000","last_undegraded":"2024-03-16T09:40:58.860897+0000","last_fullsized":"2024-03-16T09:40:58.860897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:58:44.451068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865167+0000","last_change":"2024-03-16T09:39:56.196887+0000","last_active":"2024-03-16T09:40:58.865167+0000","last_peered":"2024-03-16T09:40:58.865167+0000","last_clean":"2024-03-16T09:40:58.865167+0000","last_became_active":"2024-03-16T09:39:56.196568+0000","last_became_peered":"2024-03-16T09:39:56.196568+0000","last_unstale":"2024-03-16T09:40:58.865167+0000","last_undegraded":"2024-03-16T09:40:58.865167+0000","last_fullsized":"2024-03-16T09:40:58.865167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:08:35.577630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865027+0000","last_change":"2024-03-16T09:39:56.186394+0000","last_active":"2024-03-16T09:40:58.865027+0000","last_peered":"2024-03-16T09:40:58.865027+0000","last_clean":"2024-03-16T09:40:58.865027+0000","last_became_active":"2024-03-16T09:39:56.186053+0000","last_became_peered":"2024-03-16T09:39:56.186053+0000","last_unstale":"2024-03-16T09:40:58.865027+0000","last_undegraded":"2024-03-16T09:40:58.865027+0000","last_fullsized":"2024-03-16T09:40:58.865027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:41.529601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865405+0000","last_change":"2024-03-16T09:39:56.197622+0000","last_active":"2024-03-16T09:40:58.865405+0000","last_peered":"2024-03-16T09:40:58.865405+0000","last_clean":"2024-03-16T09:40:58.865405+0000","last_became_active":"2024-03-16T09:39:56.196885+0000","last_became_peered":"2024-03-16T09:39:56.196885+0000","last_unstale":"2024-03-16T09:40:58.865405+0000","last_undegraded":"2024-03-16T09:40:58.865405+0000","last_fullsized":"2024-03-16T09:40:58.865405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:08:50.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.868417+0000","last_change":"2024-03-16T09:40:58.868417+0000","last_active":"2024-03-16T09:40:58.868417+0000","last_peered":"2024-03-16T09:40:58.868417+0000","last_clean":"2024-03-16T09:40:58.868417+0000","last_became_active":"2024-03-16T09:40:58.868100+0000","last_became_peered":"2024-03-16T09:40:58.868100+0000","last_unstale":"2024-03-16T09:40:58.868417+0000","last_undegraded":"2024-03-16T09:40:58.868417+0000","last_fullsized":"2024-03-16T09:40:58.868417+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:51.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.861123+0000","last_change":"2024-03-16T09:39:56.181262+0000","last_active":"2024-03-16T09:40:58.861123+0000","last_peered":"2024-03-16T09:40:58.861123+0000","last_clean":"2024-03-16T09:40:58.861123+0000","last_became_active":"2024-03-16T09:39:56.180916+0000","last_became_peered":"2024-03-16T09:39:56.180916+0000","last_unstale":"2024-03-16T09:40:58.861123+0000","last_undegraded":"2024-03-16T09:40:58.861123+0000","last_fullsized":"2024-03-16T09:40:58.861123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:34:45.056025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865776+0000","last_change":"2024-03-16T09:39:56.176621+0000","last_active":"2024-03-16T09:40:58.865776+0000","last_peered":"2024-03-16T09:40:58.865776+0000","last_clean":"2024-03-16T09:40:58.865776+0000","last_became_active":"2024-03-16T09:39:56.176296+0000","last_became_peered":"2024-03-16T09:39:56.176296+0000","last_unstale":"2024-03-16T09:40:58.865776+0000","last_undegraded":"2024-03-16T09:40:58.865776+0000","last_fullsized":"2024-03-16T09:40:58.865776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:54.673937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871473+0000","last_change":"2024-03-16T09:39:56.197689+0000","last_active":"2024-03-16T09:40:58.871473+0000","last_peered":"2024-03-16T09:40:58.871473+0000","last_clean":"2024-03-16T09:40:58.871473+0000","last_became_active":"2024-03-16T09:39:56.197343+0000","last_became_peered":"2024-03-16T09:39:56.197343+0000","last_unstale":"2024-03-16T09:40:58.871473+0000","last_undegraded":"2024-03-16T09:40:58.871473+0000","last_fullsized":"2024-03-16T09:40:58.871473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:28:10.218048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.862904+0000","last_change":"2024-03-16T09:39:56.193551+0000","last_active":"2024-03-16T09:40:58.862904+0000","last_peered":"2024-03-16T09:40:58.862904+0000","last_clean":"2024-03-16T09:40:58.862904+0000","last_became_active":"2024-03-16T09:39:56.192901+0000","last_became_peered":"2024-03-16T09:39:56.192901+0000","last_unstale":"2024-03-16T09:40:58.862904+0000","last_undegraded":"2024-03-16T09:40:58.862904+0000","last_fullsized":"2024-03-16T09:40:58.862904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:50:15.528370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871264+0000","last_change":"2024-03-16T09:39:56.188484+0000","last_active":"2024-03-16T09:40:58.871264+0000","last_peered":"2024-03-16T09:40:58.871264+0000","last_clean":"2024-03-16T09:40:58.871264+0000","last_became_active":"2024-03-16T09:39:56.188144+0000","last_became_peered":"2024-03-16T09:39:56.188144+0000","last_unstale":"2024-03-16T09:40:58.871264+0000","last_undegraded":"2024-03-16T09:40:58.871264+0000","last_fullsized":"2024-03-16T09:40:58.871264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:01:10.042765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863200+0000","last_change":"2024-03-16T09:39:56.187448+0000","last_active":"2024-03-16T09:40:58.863200+0000","last_peered":"2024-03-16T09:40:58.863200+0000","last_clean":"2024-03-16T09:40:58.863200+0000","last_became_active":"2024-03-16T09:39:56.186772+0000","last_became_peered":"2024-03-16T09:39:56.186772+0000","last_unstale":"2024-03-16T09:40:58.863200+0000","last_undegraded":"2024-03-16T09:40:58.863200+0000","last_fullsized":"2024-03-16T09:40:58.863200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:45:43.315011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.866205+0000","last_change":"2024-03-16T09:39:56.191004+0000","last_active":"2024-03-16T09:40:58.866205+0000","last_peered":"2024-03-16T09:40:58.866205+0000","last_clean":"2024-03-16T09:40:58.866205+0000","last_became_active":"2024-03-16T09:39:56.190646+0000","last_became_peered":"2024-03-16T09:39:56.190646+0000","last_unstale":"2024-03-16T09:40:58.866205+0000","last_undegraded":"2024-03-16T09:40:58.866205+0000","last_fullsized":"2024-03-16T09:40:58.866205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:03.191129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860886+0000","last_change":"2024-03-16T09:39:56.197552+0000","last_active":"2024-03-16T09:40:58.860886+0000","last_peered":"2024-03-16T09:40:58.860886+0000","last_clean":"2024-03-16T09:40:58.860886+0000","last_became_active":"2024-03-16T09:39:56.197215+0000","last_became_peered":"2024-03-16T09:39:56.197215+0000","last_unstale":"2024-03-16T09:40:58.860886+0000","last_undegraded":"2024-03-16T09:40:58.860886+0000","last_fullsized":"2024-03-16T09:40:58.860886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:45.795328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63633,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":448,"num_read_kb":293,"num_write":155,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":460713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1650688,"data_stored":1112195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":460825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1564672,"data_stored":1052386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":460316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":520132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":461587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":521482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":460627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":460753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36884,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708268,"statfs":{"total":95995035648,"available":95957266432,"internally_reserved":0,"allocated":1576960,"data_stored":1052070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T09:41:12.523 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-16T09:41:12.523 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T09:41:12.523 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-16T09:41:12.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-16T09:41:12.763 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:13.114 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:12 smithi077 ceph-mon[32171]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T09:41:13.114 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:12 smithi077 ceph-mon[32171]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:12 smithi119 ceph-mon[35632]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T09:41:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:12 smithi119 ceph-mon[35632]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:13.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:12 smithi129 ceph-mon[35841]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T09:41:13.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:12 smithi129 ceph-mon[35841]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 -- 172.21.15.77:0/1452722245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084074a40 msgr2=0x7f2084074ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 --2- 172.21.15.77:0/1452722245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084074a40 0x7f2084074ec0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2070009990 tx=0x7f2070009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 -- 172.21.15.77:0/1452722245 shutdown_connections 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 --2- 172.21.15.77:0/1452722245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2084076300 0x7f2084074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 --2- 172.21.15.77:0/1452722245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2084112f60 0x7f2084115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 --2- 172.21.15.77:0/1452722245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084074a40 0x7f2084074ec0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 -- 172.21.15.77:0/1452722245 >> 172.21.15.77:0/1452722245 conn(0x7f2084079400 msgr2=0x7f2084079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 -- 172.21.15.77:0/1452722245 shutdown_connections 2024-03-16T09:41:14.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 -- 172.21.15.77:0/1452722245 wait complete. 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 Processor -- start 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.376+0000 7f208aad2700 1 -- start start 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208aad2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2084074a40 0x7f208419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208aad2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2084076300 0x7f208419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208aad2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084112f60 0x7f20841a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208aad2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2084118a40 con 0x7f2084112f60 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208aad2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f20841188c0 con 0x7f2084074a40 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208aad2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2084118bc0 con 0x7f2084076300 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208906f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084112f60 0x7f20841a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f2083fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2084076300 0x7f208419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208886e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2084074a40 0x7f208419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208906f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084112f60 0x7f20841a38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45628/0 (socket says 172.21.15.77:45628) 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208906f700 1 -- 172.21.15.77:0/3134220316 learned_addr learned my addr 172.21.15.77:0/3134220316 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:14.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208886e700 1 -- 172.21.15.77:0/3134220316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2084076300 msgr2=0x7f208419f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208886e700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2084076300 0x7f208419f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208886e700 1 -- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084112f60 msgr2=0x7f20841a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208886e700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084112f60 0x7f20841a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208886e700 1 -- 172.21.15.77:0/3134220316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20841a4020 con 0x7f2084074a40 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208906f700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084112f60 0x7f20841a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f2083fff700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2084076300 0x7f208419f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208886e700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2084074a40 0x7f208419f010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f207800e9f0 tx=0x7f207800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:14.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f2081ffb700 1 -- 172.21.15.77:0/3134220316 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207800c9c0 con 0x7f2084074a40 2024-03-16T09:41:14.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20841a4310 con 0x7f2084074a40 2024-03-16T09:41:14.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f2081ffb700 1 -- 172.21.15.77:0/3134220316 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f207800cb60 con 0x7f2084074a40 2024-03-16T09:41:14.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.377+0000 7f2081ffb700 1 -- 172.21.15.77:0/3134220316 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078018680 con 0x7f2084074a40 2024-03-16T09:41:14.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.378+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20841abb90 con 0x7f2084074a40 2024-03-16T09:41:14.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.378+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2084068f90 con 0x7f2084074a40 2024-03-16T09:41:14.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.379+0000 7f2081ffb700 1 -- 172.21.15.77:0/3134220316 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2078047a90 con 0x7f2084074a40 2024-03-16T09:41:14.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.380+0000 7f2081ffb700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f20740a2920 0x7f20740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:14.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.380+0000 7f2083fff700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f20740a2920 0x7f20740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:14.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.380+0000 7f2081ffb700 1 -- 172.21.15.77:0/3134220316 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f20780cee10 con 0x7f2084074a40 2024-03-16T09:41:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.380+0000 7f2083fff700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f20740a2920 0x7f20740a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2084079d90 tx=0x7f207001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.384+0000 7f2081ffb700 1 -- 172.21.15.77:0/3134220316 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f207809bd00 con 0x7f2084074a40 2024-03-16T09:41:14.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.623+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f20841abdc0 con 0x7f2084074a40 2024-03-16T09:41:14.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.623+0000 7f2081ffb700 1 -- 172.21.15.77:0/3134220316 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f207809b600 con 0x7f2084074a40 2024-03-16T09:41:14.626 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:14.626 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T09:41:14.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f20740a2920 msgr2=0x7f20740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:14.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f20740a2920 0x7f20740a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2084079d90 tx=0x7f207001a040 comp rx=0 tx=0).stop 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2084074a40 msgr2=0x7f208419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2084074a40 0x7f208419f010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f207800e9f0 tx=0x7f207800ed00 comp rx=0 tx=0).stop 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 shutdown_connections 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f20740a2920 0x7f20740a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2084074a40 0x7f208419f010 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.625+0000 7f208aad2700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2084112f60 0x7f20841a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.626+0000 7f208aad2700 1 --2- 172.21.15.77:0/3134220316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2084076300 0x7f208419f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.626+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 >> 172.21.15.77:0/3134220316 conn(0x7f2084079400 msgr2=0x7f20841117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.626+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 shutdown_connections 2024-03-16T09:41:14.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:14.626+0000 7f208aad2700 1 -- 172.21.15.77:0/3134220316 wait complete. 2024-03-16T09:41:15.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:14 smithi077 ceph-mon[32171]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T09:41:15.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:14 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3134220316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T09:41:15.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:14 smithi119 ceph-mon[35632]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T09:41:15.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:14 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3134220316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T09:41:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:14 smithi129 ceph-mon[35841]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 objects/s recovering 2024-03-16T09:41:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:14 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3134220316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T09:41:15.351 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-16T09:41:15.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:15.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.455+0000 7f9060f1b640 1 -- 172.21.15.77:0/19313432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c100700 msgr2=0x7f905c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.455+0000 7f9060f1b640 1 --2- 172.21.15.77:0/19313432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c100700 0x7f905c100b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9054009900 tx=0x7f905402bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 -- 172.21.15.77:0/19313432 shutdown_connections 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 --2- 172.21.15.77:0/19313432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f905c1022b0 0x7f905c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 --2- 172.21.15.77:0/19313432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f905c1018f0 0x7f905c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 --2- 172.21.15.77:0/19313432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c100700 0x7f905c100b20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 -- 172.21.15.77:0/19313432 >> 172.21.15.77:0/19313432 conn(0x7f905c0fbc70 msgr2=0x7f905c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 -- 172.21.15.77:0/19313432 shutdown_connections 2024-03-16T09:41:15.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 -- 172.21.15.77:0/19313432 wait complete. 2024-03-16T09:41:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 Processor -- start 2024-03-16T09:41:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 -- start start 2024-03-16T09:41:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f905c100700 0x7f905c104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c1018f0 0x7f905c102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f905c1022b0 0x7f905c103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10c230 con 0x7f905c1018f0 2024-03-16T09:41:15.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.456+0000 7f9060f1b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10c0b0 con 0x7f905c1022b0 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9060f1b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10c3b0 con 0x7f905c100700 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c1018f0 0x7f905c102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f905a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f905c100700 0x7f905c104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c1018f0 0x7f905c102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52708/0 (socket says 172.21.15.77:52708) 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 -- 172.21.15.77:0/544272307 learned_addr learned my addr 172.21.15.77:0/544272307 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f905ad9d640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f905c1022b0 0x7f905c103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 -- 172.21.15.77:0/544272307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f905c100700 msgr2=0x7f905c104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f905c100700 0x7f905c104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 -- 172.21.15.77:0/544272307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f905c1022b0 msgr2=0x7f905c103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f905c1022b0 0x7f905c103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 -- 172.21.15.77:0/544272307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c103aa0 con 0x7f905c1018f0 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9059d9b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c1018f0 0x7f905c102cd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f905000ebc0 tx=0x7f905000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f904b7fe640 1 -- 172.21.15.77:0/544272307 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050010630 con 0x7f905c1018f0 2024-03-16T09:41:15.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c1a1ba0 con 0x7f905c1018f0 2024-03-16T09:41:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f904b7fe640 1 -- 172.21.15.77:0/544272307 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9050010c80 con 0x7f905c1018f0 2024-03-16T09:41:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f904b7fe640 1 -- 172.21.15.77:0/544272307 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050018b30 con 0x7f905c1018f0 2024-03-16T09:41:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.457+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1a2140 con 0x7f905c1018f0 2024-03-16T09:41:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.458+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f905c1a2310 con 0x7f905c1018f0 2024-03-16T09:41:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.459+0000 7f904b7fe640 1 -- 172.21.15.77:0/544272307 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9050018cd0 con 0x7f905c1018f0 2024-03-16T09:41:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.460+0000 7f904b7fe640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f90440a2890 0x7f90440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.460+0000 7f905a59c640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f90440a2890 0x7f90440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.460+0000 7f904b7fe640 1 -- 172.21.15.77:0/544272307 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f90500cf660 con 0x7f905c1018f0 2024-03-16T09:41:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.460+0000 7f905a59c640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f90440a2890 0x7f90440a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9054005ab0 tx=0x7f9054005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.464+0000 7f904b7fe640 1 -- 172.21.15.77:0/544272307 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f905009bf60 con 0x7f905c1018f0 2024-03-16T09:41:15.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.647+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f905c1a2600 con 0x7f905c1018f0 2024-03-16T09:41:15.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.649+0000 7f904b7fe640 1 -- 172.21.15.77:0/544272307 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f90500a0000 con 0x7f905c1018f0 2024-03-16T09:41:15.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:15.652 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f90440a2890 msgr2=0x7f90440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f90440a2890 0x7f90440a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9054005ab0 tx=0x7f9054005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c1018f0 msgr2=0x7f905c102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c1018f0 0x7f905c102cd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f905000ebc0 tx=0x7f905000c6a0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 shutdown_connections 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f90440a2890 0x7f90440a4d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f905c1022b0 0x7f905c103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f905c1018f0 0x7f905c102cd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 --2- 172.21.15.77:0/544272307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f905c100700 0x7f905c104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 >> 172.21.15.77:0/544272307 conn(0x7f905c0fbc70 msgr2=0x7f905c068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:15.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 shutdown_connections 2024-03-16T09:41:15.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.651+0000 7f9060f1b640 1 -- 172.21.15.77:0/544272307 wait complete. 2024-03-16T09:41:15.665 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:41:15.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:41:15.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- 172.21.15.77:0/3042395918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef41033c0 msgr2=0x7f5ef4109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 --2- 172.21.15.77:0/3042395918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef41033c0 0x7f5ef4109c50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009900 tx=0x7f5ef002bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:15.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- 172.21.15.77:0/3042395918 shutdown_connections 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 --2- 172.21.15.77:0/3042395918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef41033c0 0x7f5ef4109c50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 --2- 172.21.15.77:0/3042395918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ef4102a00 0x7f5ef4102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 --2- 172.21.15.77:0/3042395918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef4101810 0x7f5ef4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- 172.21.15.77:0/3042395918 >> 172.21.15.77:0/3042395918 conn(0x7f5ef40fcd80 msgr2=0x7f5ef40ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- 172.21.15.77:0/3042395918 shutdown_connections 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- 172.21.15.77:0/3042395918 wait complete. 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 Processor -- start 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- start start 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ef4101810 0x7f5ef4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef4102a00 0x7f5ef4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef41033c0 0x7f5ef419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef410d340 con 0x7f5ef4102a00 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef410d1c0 con 0x7f5ef4101810 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.772+0000 7f5efbaac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef410d4c0 con 0x7f5ef41033c0 2024-03-16T09:41:15.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5ef9047640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef4102a00 0x7f5ef4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5ef9047640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef4102a00 0x7f5ef4197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52720/0 (socket says 172.21.15.77:52720) 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5ef9047640 1 -- 172.21.15.77:0/4054833954 learned_addr learned my addr 172.21.15.77:0/4054833954 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efa049640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef41033c0 0x7f5ef419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5ef9848640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ef4101810 0x7f5ef4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efa049640 1 -- 172.21.15.77:0/4054833954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ef4101810 msgr2=0x7f5ef4197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efa049640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ef4101810 0x7f5ef4197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efa049640 1 -- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef4102a00 msgr2=0x7f5ef4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efa049640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef4102a00 0x7f5ef4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efa049640 1 -- 172.21.15.77:0/4054833954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef419f640 con 0x7f5ef41033c0 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5ef9047640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef4102a00 0x7f5ef4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5ef9848640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ef4101810 0x7f5ef4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efa049640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef41033c0 0x7f5ef419efd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009230 tx=0x7f5ef0005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5eeaffd640 1 -- 172.21.15.77:0/4054833954 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0043020 con 0x7f5ef41033c0 2024-03-16T09:41:15.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef419f8d0 con 0x7f5ef41033c0 2024-03-16T09:41:15.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5eeaffd640 1 -- 172.21.15.77:0/4054833954 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef00043d0 con 0x7f5ef41033c0 2024-03-16T09:41:15.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5eeaffd640 1 -- 172.21.15.77:0/4054833954 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0003aa0 con 0x7f5ef41033c0 2024-03-16T09:41:15.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.773+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef419fe70 con 0x7f5ef41033c0 2024-03-16T09:41:15.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.774+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef4108b50 con 0x7f5ef41033c0 2024-03-16T09:41:15.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.775+0000 7f5eeaffd640 1 -- 172.21.15.77:0/4054833954 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5ef0046020 con 0x7f5ef41033c0 2024-03-16T09:41:15.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.776+0000 7f5eeaffd640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee00a2890 0x7f5ee00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:15.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.776+0000 7f5ef9848640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee00a2890 0x7f5ee00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:15.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.776+0000 7f5eeaffd640 1 -- 172.21.15.77:0/4054833954 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f5ef00f0c50 con 0x7f5ef41033c0 2024-03-16T09:41:15.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.776+0000 7f5ef9848640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee00a2890 0x7f5ee00a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4102860 tx=0x7f5eec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.781+0000 7f5eeaffd640 1 -- 172.21.15.77:0/4054833954 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ef004d020 con 0x7f5ef41033c0 2024-03-16T09:41:15.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.964+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5ef402d250 con 0x7f5ef41033c0 2024-03-16T09:41:15.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.965+0000 7f5eeaffd640 1 -- 172.21.15.77:0/4054833954 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f5ef00bdcb0 con 0x7f5ef41033c0 2024-03-16T09:41:15.968 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:41:15.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee00a2890 msgr2=0x7f5ee00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee00a2890 0x7f5ee00a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4102860 tx=0x7f5eec009500 comp rx=0 tx=0).stop 2024-03-16T09:41:15.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef41033c0 msgr2=0x7f5ef419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:15.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef41033c0 0x7f5ef419efd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009230 tx=0x7f5ef0005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:15.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 shutdown_connections 2024-03-16T09:41:15.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee00a2890 0x7f5ee00a4d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef41033c0 0x7f5ef419efd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ef4102a00 0x7f5ef4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 --2- 172.21.15.77:0/4054833954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ef4101810 0x7f5ef4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 >> 172.21.15.77:0/4054833954 conn(0x7f5ef40fcd80 msgr2=0x7f5ef4105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 shutdown_connections 2024-03-16T09:41:15.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:15.967+0000 7f5efbaac640 1 -- 172.21.15.77:0/4054833954 wait complete. 2024-03-16T09:41:15.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:15 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/544272307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:15.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 -- 172.21.15.77:0/2453889970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6284102290 msgr2=0x7f6284108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 --2- 172.21.15.77:0/2453889970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6284102290 0x7f6284108b20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6274009900 tx=0x7f627402bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 -- 172.21.15.77:0/2453889970 shutdown_connections 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 --2- 172.21.15.77:0/2453889970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6284102290 0x7f6284108b20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 --2- 172.21.15.77:0/2453889970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62841018d0 0x7f6284101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 --2- 172.21.15.77:0/2453889970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62841006e0 0x7f6284100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 -- 172.21.15.77:0/2453889970 >> 172.21.15.77:0/2453889970 conn(0x7f62840fbc70 msgr2=0x7f62840fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 -- 172.21.15.77:0/2453889970 shutdown_connections 2024-03-16T09:41:16.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 -- 172.21.15.77:0/2453889970 wait complete. 2024-03-16T09:41:16.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.086+0000 7f62899d4640 1 Processor -- start 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 -- start start 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62841006e0 0x7f628410b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62841018d0 0x7f62841096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6284102290 0x7f6284109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f628410d330 con 0x7f62841018d0 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f628410d1b0 con 0x7f62841006e0 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f628410d4b0 con 0x7f6284102290 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62841018d0 0x7f62841096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62841018d0 0x7f62841096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52726/0 (socket says 172.21.15.77:52726) 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 -- 172.21.15.77:0/169502154 learned_addr learned my addr 172.21.15.77:0/169502154 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f6282ffd640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62841006e0 0x7f628410b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62837fe640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6284102290 0x7f6284109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 -- 172.21.15.77:0/169502154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6284102290 msgr2=0x7f6284109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6284102290 0x7f6284109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 -- 172.21.15.77:0/169502154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62841006e0 msgr2=0x7f628410b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62841006e0 0x7f628410b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 -- 172.21.15.77:0/169502154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628410a470 con 0x7f62841018d0 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62827fc640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62841018d0 0x7f62841096a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f627800ebc0 tx=0x7f627800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62889d2640 1 -- 172.21.15.77:0/169502154 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278010630 con 0x7f62841018d0 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62841a1b30 con 0x7f62841018d0 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62889d2640 1 -- 172.21.15.77:0/169502154 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6278010c80 con 0x7f62841018d0 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.087+0000 7f62889d2640 1 -- 172.21.15.77:0/169502154 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278018b30 con 0x7f62841018d0 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.088+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62841a20d0 con 0x7f62841018d0 2024-03-16T09:41:16.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.088+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6284107a70 con 0x7f62841018d0 2024-03-16T09:41:16.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.090+0000 7f62889d2640 1 -- 172.21.15.77:0/169502154 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6278018cd0 con 0x7f62841018d0 2024-03-16T09:41:16.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.090+0000 7f62889d2640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f626c0a2890 0x7f626c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.090+0000 7f6282ffd640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f626c0a2890 0x7f626c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.090+0000 7f62889d2640 1 -- 172.21.15.77:0/169502154 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f62780cf660 con 0x7f62841018d0 2024-03-16T09:41:16.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.091+0000 7f6282ffd640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f626c0a2890 0x7f626c0a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6284101730 tx=0x7f627c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:16.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.098+0000 7f62889d2640 1 -- 172.21.15.77:0/169502154 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f627809bf60 con 0x7f62841018d0 2024-03-16T09:41:16.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:15 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/544272307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:16.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.280+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f628402d1b0 con 0x7f62841018d0 2024-03-16T09:41:16.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.280+0000 7f62889d2640 1 -- 172.21.15.77:0/169502154 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f62780a0000 con 0x7f62841018d0 2024-03-16T09:41:16.283 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.282+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f626c0a2890 msgr2=0x7f626c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f626c0a2890 0x7f626c0a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6284101730 tx=0x7f627c005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62841018d0 msgr2=0x7f62841096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62841018d0 0x7f62841096a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f627800ebc0 tx=0x7f627800c6a0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 shutdown_connections 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f626c0a2890 0x7f626c0a4d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6284102290 0x7f6284109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62841018d0 0x7f62841096a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 --2- 172.21.15.77:0/169502154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62841006e0 0x7f628410b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 >> 172.21.15.77:0/169502154 conn(0x7f62840fbc70 msgr2=0x7f6284104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:16.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 shutdown_connections 2024-03-16T09:41:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.283+0000 7f62899d4640 1 -- 172.21.15.77:0/169502154 wait complete. 2024-03-16T09:41:16.296 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:41:16.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:15 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/544272307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:16.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 -- 172.21.15.77:0/1391943429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 msgr2=0x7f2ab8106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 --2- 172.21.15.77:0/1391943429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8106f80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009900 tx=0x7f2aa402bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:16.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 -- 172.21.15.77:0/1391943429 shutdown_connections 2024-03-16T09:41:16.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 --2- 172.21.15.77:0/1391943429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ab81074c0 0x7f2ab8109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 --2- 172.21.15.77:0/1391943429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8106f80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 --2- 172.21.15.77:0/1391943429 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8102260 0x7f2ab8104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 -- 172.21.15.77:0/1391943429 >> 172.21.15.77:0/1391943429 conn(0x7f2ab80fbc70 msgr2=0x7f2ab80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 -- 172.21.15.77:0/1391943429 shutdown_connections 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.398+0000 7f2abee7e640 1 -- 172.21.15.77:0/1391943429 wait complete. 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 Processor -- start 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 -- start start 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8102260 0x7f2ab8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ab81074c0 0x7f2ab819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810d310 con 0x7f2ab8104b90 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810d190 con 0x7f2ab8102260 2024-03-16T09:41:16.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abee7e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810d490 con 0x7f2ab81074c0 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52752/0 (socket says 172.21.15.77:52752) 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2ab7fff640 1 -- 172.21.15.77:0/4240592021 learned_addr learned my addr 172.21.15.77:0/4240592021 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abcc1a640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8102260 0x7f2ab8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.399+0000 7f2abd41b640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ab81074c0 0x7f2ab819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab7fff640 1 -- 172.21.15.77:0/4240592021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ab81074c0 msgr2=0x7f2ab819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab7fff640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ab81074c0 0x7f2ab819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab7fff640 1 -- 172.21.15.77:0/4240592021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8102260 msgr2=0x7f2ab8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab7fff640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8102260 0x7f2ab8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab7fff640 1 -- 172.21.15.77:0/4240592021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab819f6b0 con 0x7f2ab8104b90 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab7fff640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8197fd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000c00 tx=0x7f2aa4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab5ffb640 1 -- 172.21.15.77:0/4240592021 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4043020 con 0x7f2ab8104b90 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab819f940 con 0x7f2ab8104b90 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab5ffb640 1 -- 172.21.15.77:0/4240592021 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2aa4004350 con 0x7f2ab8104b90 2024-03-16T09:41:16.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2ab5ffb640 1 -- 172.21.15.77:0/4240592021 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4003b80 con 0x7f2ab8104b90 2024-03-16T09:41:16.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.400+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab819fee0 con 0x7f2ab8104b90 2024-03-16T09:41:16.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.401+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab80fd760 con 0x7f2ab8104b90 2024-03-16T09:41:16.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.403+0000 7f2ab5ffb640 1 -- 172.21.15.77:0/4240592021 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2aa4046020 con 0x7f2ab8104b90 2024-03-16T09:41:16.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.403+0000 7f2ab5ffb640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aa80a2890 0x7f2aa80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.403+0000 7f2ab5ffb640 1 -- 172.21.15.77:0/4240592021 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2aa40f0bd0 con 0x7f2ab8104b90 2024-03-16T09:41:16.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.403+0000 7f2abcc1a640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aa80a2890 0x7f2aa80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.404+0000 7f2abcc1a640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aa80a2890 0x7f2aa80a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009bd0 tx=0x7f2aac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.408+0000 7f2ab5ffb640 1 -- 172.21.15.77:0/4240592021 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2aa404d020 con 0x7f2ab8104b90 2024-03-16T09:41:16.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.583+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2ab802d360 con 0x7f2ab8104b90 2024-03-16T09:41:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.583+0000 7f2ab5ffb640 1 -- 172.21.15.77:0/4240592021 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f2aa40bdc30 con 0x7f2ab8104b90 2024-03-16T09:41:16.586 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aa80a2890 msgr2=0x7f2aa80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aa80a2890 0x7f2aa80a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009bd0 tx=0x7f2aac005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 msgr2=0x7f2ab8197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8197fd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000c00 tx=0x7f2aa4005dc0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 shutdown_connections 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2aa80a2890 0x7f2aa80a4d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ab81074c0 0x7f2ab819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ab8104b90 0x7f2ab8197fd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 --2- 172.21.15.77:0/4240592021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8102260 0x7f2ab8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 >> 172.21.15.77:0/4240592021 conn(0x7f2ab80fbc70 msgr2=0x7f2ab81040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:16.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 shutdown_connections 2024-03-16T09:41:16.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.586+0000 7f2abee7e640 1 -- 172.21.15.77:0/4240592021 wait complete. 2024-03-16T09:41:16.602 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 -- 172.21.15.77:0/684074077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b701033e0 msgr2=0x7f8b70109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 --2- 172.21.15.77:0/684074077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b701033e0 0x7f8b70109c70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009ca0 tx=0x7f8b6802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 -- 172.21.15.77:0/684074077 shutdown_connections 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 --2- 172.21.15.77:0/684074077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b701033e0 0x7f8b70109c70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 --2- 172.21.15.77:0/684074077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b70102a20 0x7f8b70102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 --2- 172.21.15.77:0/684074077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b70101830 0x7f8b70101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 -- 172.21.15.77:0/684074077 >> 172.21.15.77:0/684074077 conn(0x7f8b700fcd80 msgr2=0x7f8b700ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:16.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.700+0000 7f8b75dab640 1 -- 172.21.15.77:0/684074077 shutdown_connections 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 -- 172.21.15.77:0/684074077 wait complete. 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 Processor -- start 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 -- start start 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b70101830 0x7f8b70197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b70102a20 0x7f8b70197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b701033e0 0x7f8b7019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7010d360 con 0x7f8b70101830 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7010d1e0 con 0x7f8b701033e0 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b75dab640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7010d4e0 con 0x7f8b70102a20 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b70101830 0x7f8b70197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b70101830 0x7f8b70197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52762/0 (socket says 172.21.15.77:52762) 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b6f7fe640 1 -- 172.21.15.77:0/2411699426 learned_addr learned my addr 172.21.15.77:0/2411699426 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.701+0000 7f8b6effd640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b70102a20 0x7f8b70197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6ffff640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b701033e0 0x7f8b7019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6f7fe640 1 -- 172.21.15.77:0/2411699426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b70102a20 msgr2=0x7f8b70197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6f7fe640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b70102a20 0x7f8b70197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6f7fe640 1 -- 172.21.15.77:0/2411699426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b701033e0 msgr2=0x7f8b7019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6f7fe640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b701033e0 0x7f8b7019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6f7fe640 1 -- 172.21.15.77:0/2411699426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7019f620 con 0x7f8b70101830 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6ffff640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b701033e0 0x7f8b7019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6f7fe640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b70101830 0x7f8b70197a00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000e9c0 tx=0x7f8b6000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6cff9640 1 -- 172.21.15.77:0/2411699426 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000cd80 con 0x7f8b70101830 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6cff9640 1 -- 172.21.15.77:0/2411699426 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b600035f0 con 0x7f8b70101830 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7019f910 con 0x7f8b70101830 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b6cff9640 1 -- 172.21.15.77:0/2411699426 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60010690 con 0x7f8b70101830 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.702+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b7019fe60 con 0x7f8b70101830 2024-03-16T09:41:16.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.703+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b70108bd0 con 0x7f8b70101830 2024-03-16T09:41:16.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.704+0000 7f8b6cff9640 1 -- 172.21.15.77:0/2411699426 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8b600037a0 con 0x7f8b70101830 2024-03-16T09:41:16.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.705+0000 7f8b6cff9640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b580a2910 0x7f8b580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:16.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.705+0000 7f8b6effd640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b580a2910 0x7f8b580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:16.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.705+0000 7f8b6cff9640 1 -- 172.21.15.77:0/2411699426 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8b600cf130 con 0x7f8b70101830 2024-03-16T09:41:16.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.705+0000 7f8b6effd640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b580a2910 0x7f8b580a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8b70198f10 tx=0x7f8b64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:16.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.709+0000 7f8b6cff9640 1 -- 172.21.15.77:0/2411699426 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b6009c080 con 0x7f8b70101830 2024-03-16T09:41:16.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.926+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b7002d1b0 con 0x7f8b70101830 2024-03-16T09:41:16.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.926+0000 7f8b6cff9640 1 -- 172.21.15.77:0/2411699426 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11878 (secure 0 0 0) 0x7f8b6009b980 con 0x7f8b70101830 2024-03-16T09:41:16.929 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:16.929 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:39:59.725431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24545,24503]},"id":1}]} 2024-03-16T09:41:16.930 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-03-16T09:41:16.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b580a2910 msgr2=0x7f8b580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b580a2910 0x7f8b580a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8b70198f10 tx=0x7f8b64005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:16.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b70101830 msgr2=0x7f8b70197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:16.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b70101830 0x7f8b70197a00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000e9c0 tx=0x7f8b6000ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:16.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 shutdown_connections 2024-03-16T09:41:16.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b580a2910 0x7f8b580a4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b701033e0 0x7f8b7019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b70102a20 0x7f8b70197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 --2- 172.21.15.77:0/2411699426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b70101830 0x7f8b70197a00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:16.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 >> 172.21.15.77:0/2411699426 conn(0x7f8b700fcd80 msgr2=0x7f8b70105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:16.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.928+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 shutdown_connections 2024-03-16T09:41:16.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:16.929+0000 7f8b75dab640 1 -- 172.21.15.77:0/2411699426 wait complete. 2024-03-16T09:41:16.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 -- 172.21.15.77:0/3839471552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb00ffae0 msgr2=0x7effb00fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 --2- 172.21.15.77:0/3839471552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb00ffae0 0x7effb00fff00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7effac009930 tx=0x7effac02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 -- 172.21.15.77:0/3839471552 shutdown_connections 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 --2- 172.21.15.77:0/3839471552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb0103470 0x7effb0109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 --2- 172.21.15.77:0/3839471552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb0102ab0 0x7effb0102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 --2- 172.21.15.77:0/3839471552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb00ffae0 0x7effb00fff00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 -- 172.21.15.77:0/3839471552 >> 172.21.15.77:0/3839471552 conn(0x7effb00fb410 msgr2=0x7effb00fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 -- 172.21.15.77:0/3839471552 shutdown_connections 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.041+0000 7effb753b640 1 -- 172.21.15.77:0/3839471552 wait complete. 2024-03-16T09:41:17.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 Processor -- start 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 -- start start 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb00ffae0 0x7effb0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb0102ab0 0x7effb0197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb0103470 0x7effb019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effb010d340 con 0x7effb0102ab0 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7effb010d1c0 con 0x7effb00ffae0 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb753b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7effb010d4c0 con 0x7effb0103470 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb0102ab0 0x7effb0197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb0102ab0 0x7effb0197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52786/0 (socket says 172.21.15.77:52786) 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 -- 172.21.15.77:0/134822666 learned_addr learned my addr 172.21.15.77:0/134822666 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:17.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb52d7640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb00ffae0 0x7effb0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb5ad8640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb0103470 0x7effb019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 -- 172.21.15.77:0/134822666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb0103470 msgr2=0x7effb019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb0103470 0x7effb019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 -- 172.21.15.77:0/134822666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb00ffae0 msgr2=0x7effb0197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb00ffae0 0x7effb0197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 -- 172.21.15.77:0/134822666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb019f690 con 0x7effb0102ab0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.042+0000 7effb4ad6640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb0102ab0 0x7effb0197fb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7effa000d950 tx=0x7effa000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.043+0000 7effa67fc640 1 -- 172.21.15.77:0/134822666 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0017070 con 0x7effb0102ab0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.043+0000 7effa67fc640 1 -- 172.21.15.77:0/134822666 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effa0009c30 con 0x7effb0102ab0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.043+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb019f980 con 0x7effb0102ab0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.043+0000 7effa67fc640 1 -- 172.21.15.77:0/134822666 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa000f620 con 0x7effb0102ab0 2024-03-16T09:41:17.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.043+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb019fed0 con 0x7effb0102ab0 2024-03-16T09:41:17.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.043+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb0108d00 con 0x7effb0102ab0 2024-03-16T09:41:17.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.045+0000 7effa67fc640 1 -- 172.21.15.77:0/134822666 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7effa00109a0 con 0x7effb0102ab0 2024-03-16T09:41:17.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.045+0000 7effa67fc640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff9c0a2890 0x7eff9c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.045+0000 7effb52d7640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff9c0a2890 0x7eff9c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.045+0000 7effa67fc640 1 -- 172.21.15.77:0/134822666 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7effa00ce770 con 0x7effb0102ab0 2024-03-16T09:41:17.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.045+0000 7effb52d7640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff9c0a2890 0x7eff9c0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7effac000c00 tx=0x7effac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.049+0000 7effa67fc640 1 -- 172.21.15.77:0/134822666 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7effa0009de0 con 0x7effb0102ab0 2024-03-16T09:41:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:16 smithi119 ceph-mon[35632]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:16 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4054833954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:16 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/169502154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:16 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4240592021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.223+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7effb002d3c0 con 0x7effb0102ab0 2024-03-16T09:41:17.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.224+0000 7effa67fc640 1 -- 172.21.15.77:0/134822666 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7effa009b070 con 0x7effb0102ab0 2024-03-16T09:41:17.227 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:17.227 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:17.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff9c0a2890 msgr2=0x7eff9c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff9c0a2890 0x7eff9c0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7effac000c00 tx=0x7effac005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb0102ab0 msgr2=0x7effb0197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb0102ab0 0x7effb0197fb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7effa000d950 tx=0x7effa000de10 comp rx=0 tx=0).stop 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 shutdown_connections 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff9c0a2890 0x7eff9c0a4d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb0103470 0x7effb019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb0102ab0 0x7effb0197fb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 --2- 172.21.15.77:0/134822666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb00ffae0 0x7effb0197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.226+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 >> 172.21.15.77:0/134822666 conn(0x7effb00fb410 msgr2=0x7effb0105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.227+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 shutdown_connections 2024-03-16T09:41:17.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.227+0000 7effb753b640 1 -- 172.21.15.77:0/134822666 wait complete. 2024-03-16T09:41:17.239 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:16 smithi077 ceph-mon[32171]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:17.239 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:16 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4054833954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.239 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:16 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/169502154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.239 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:16 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4240592021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:41:17.241 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:41:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:16 smithi129 ceph-mon[35841]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:16 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4054833954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:16 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/169502154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:16 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4240592021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 -- 172.21.15.77:0/4120865086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681017f0 msgr2=0x7f4368101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 --2- 172.21.15.77:0/4120865086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681017f0 0x7f4368101c10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4358009900 tx=0x7f435802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 -- 172.21.15.77:0/4120865086 shutdown_connections 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 --2- 172.21.15.77:0/4120865086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681033a0 0x7f4368109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 --2- 172.21.15.77:0/4120865086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681029e0 0x7f4368102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 --2- 172.21.15.77:0/4120865086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681017f0 0x7f4368101c10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 -- 172.21.15.77:0/4120865086 >> 172.21.15.77:0/4120865086 conn(0x7f43680fcd80 msgr2=0x7f43680ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:17.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 -- 172.21.15.77:0/4120865086 shutdown_connections 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 -- 172.21.15.77:0/4120865086 wait complete. 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 Processor -- start 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.343+0000 7f436fb10640 1 -- start start 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681017f0 0x7f4368197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681029e0 0x7f4368197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681033a0 0x7f436819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f436810d310 con 0x7f43681029e0 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f436810d190 con 0x7f43681017f0 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f436810d490 con 0x7f43681033a0 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436d0ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681029e0 0x7f4368197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436d8ac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681017f0 0x7f4368197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681033a0 0x7f436819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681033a0 0x7f436819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:32822/0 (socket says 172.21.15.77:32822) 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 -- 172.21.15.77:0/2114091849 learned_addr learned my addr 172.21.15.77:0/2114091849 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436d0ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681029e0 0x7f4368197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52796/0 (socket says 172.21.15.77:52796) 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436d8ac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681017f0 0x7f4368197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:46230/0 (socket says 172.21.15.77:46230) 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 -- 172.21.15.77:0/2114091849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681017f0 msgr2=0x7f4368197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681017f0 0x7f4368197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 -- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681029e0 msgr2=0x7f4368197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681029e0 0x7f4368197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 -- 172.21.15.77:0/2114091849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436819f6b0 con 0x7f43681033a0 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436d8ac640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681017f0 0x7f4368197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436e0ad640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681033a0 0x7f436819f040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f436000eb10 tx=0x7f436000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f435effd640 1 -- 172.21.15.77:0/2114091849 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360004b20 con 0x7f43681033a0 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f435effd640 1 -- 172.21.15.77:0/2114091849 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4360004cc0 con 0x7f43681033a0 2024-03-16T09:41:17.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436819f9a0 con 0x7f43681033a0 2024-03-16T09:41:17.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f435effd640 1 -- 172.21.15.77:0/2114091849 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360010690 con 0x7f43681033a0 2024-03-16T09:41:17.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.344+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436819fea0 con 0x7f43681033a0 2024-03-16T09:41:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.346+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4368108ca0 con 0x7f43681033a0 2024-03-16T09:41:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.346+0000 7f435effd640 1 -- 172.21.15.77:0/2114091849 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4360005020 con 0x7f43681033a0 2024-03-16T09:41:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.347+0000 7f435effd640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f43540a2890 0x7f43540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.347+0000 7f436d8ac640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f43540a2890 0x7f43540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.347+0000 7f435effd640 1 -- 172.21.15.77:0/2114091849 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f43600cf170 con 0x7f43681033a0 2024-03-16T09:41:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.347+0000 7f436d8ac640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f43540a2890 0x7f43540a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4358005ab0 tx=0x7f4358005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.352+0000 7f435effd640 1 -- 172.21.15.77:0/2114091849 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f436009c0c0 con 0x7f43681033a0 2024-03-16T09:41:17.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.528+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f436802d360 con 0x7f43681033a0 2024-03-16T09:41:17.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.528+0000 7f435effd640 1 -- 172.21.15.77:0/2114091849 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f436009b9c0 con 0x7f43681033a0 2024-03-16T09:41:17.531 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:41:17.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.530+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f43540a2890 msgr2=0x7f43540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.530+0000 7f436fb10640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f43540a2890 0x7f43540a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4358005ab0 tx=0x7f4358005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.530+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681033a0 msgr2=0x7f436819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.530+0000 7f436fb10640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681033a0 0x7f436819f040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f436000eb10 tx=0x7f436000c2d0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 shutdown_connections 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f43540a2890 0x7f43540a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43681033a0 0x7f436819f040 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43681029e0 0x7f4368197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 --2- 172.21.15.77:0/2114091849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43681017f0 0x7f4368197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 >> 172.21.15.77:0/2114091849 conn(0x7f43680fcd80 msgr2=0x7f4368105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 shutdown_connections 2024-03-16T09:41:17.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.531+0000 7f436fb10640 1 -- 172.21.15.77:0/2114091849 wait complete. 2024-03-16T09:41:17.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:41:17.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/723621689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a81006e0 msgr2=0x7fc5a8100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/723621689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a81006e0 0x7fc5a8100b00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc598009ca0 tx=0x7fc59802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/723621689 shutdown_connections 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/723621689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a8102290 0x7fc5a8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/723621689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a81018d0 0x7fc5a8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/723621689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a81006e0 0x7fc5a8100b00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/723621689 >> 172.21.15.77:0/723621689 conn(0x7fc5a80fbc70 msgr2=0x7fc5a80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/723621689 shutdown_connections 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.645+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/723621689 wait complete. 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 Processor -- start 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 -- start start 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a81006e0 0x7fc5a8069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81018d0 0x7fc5a819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8102290 0x7fc5a819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a810c200 con 0x7fc5a8102290 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a810c080 con 0x7fc5a81018d0 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5ad7f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a810c380 con 0x7fc5a81006e0 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8102290 0x7fc5a819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8102290 0x7fc5a819ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52812/0 (socket says 172.21.15.77:52812) 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 -- 172.21.15.77:0/4049476901 learned_addr learned my addr 172.21.15.77:0/4049476901 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a67fc640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81018d0 0x7fc5a819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a6ffd640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a81006e0 0x7fc5a8069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 -- 172.21.15.77:0/4049476901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a81006e0 msgr2=0x7fc5a8069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a81006e0 0x7fc5a8069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 -- 172.21.15.77:0/4049476901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81018d0 msgr2=0x7fc5a819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81018d0 0x7fc5a819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a77fe640 1 -- 172.21.15.77:0/4049476901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a819f700 con 0x7fc5a8102290 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.646+0000 7fc5a67fc640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81018d0 0x7fc5a819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:17.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.647+0000 7fc5a77fe640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8102290 0x7fc5a819ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000e9b0 tx=0x7fc5a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.647+0000 7fc58ffff640 1 -- 172.21.15.77:0/4049476901 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a000cd80 con 0x7fc5a8102290 2024-03-16T09:41:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.647+0000 7fc58ffff640 1 -- 172.21.15.77:0/4049476901 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5a00035f0 con 0x7fc5a8102290 2024-03-16T09:41:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.647+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a819f9f0 con 0x7fc5a8102290 2024-03-16T09:41:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.647+0000 7fc58ffff640 1 -- 172.21.15.77:0/4049476901 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0010690 con 0x7fc5a8102290 2024-03-16T09:41:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.647+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a819fef0 con 0x7fc5a8102290 2024-03-16T09:41:17.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.648+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a8107bc0 con 0x7fc5a8102290 2024-03-16T09:41:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.649+0000 7fc58ffff640 1 -- 172.21.15.77:0/4049476901 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc5a00037a0 con 0x7fc5a8102290 2024-03-16T09:41:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.649+0000 7fc58ffff640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2890 0x7fc5900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.649+0000 7fc5a6ffd640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2890 0x7fc5900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.649+0000 7fc58ffff640 1 -- 172.21.15.77:0/4049476901 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fc5a00cef00 con 0x7fc5a8102290 2024-03-16T09:41:17.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.650+0000 7fc5a6ffd640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2890 0x7fc5900a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc598032380 tx=0x7fc598000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.654+0000 7fc58ffff640 1 -- 172.21.15.77:0/4049476901 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc5a009be50 con 0x7fc5a8102290 2024-03-16T09:41:17.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.828+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc5a80623e0 con 0x7fc5a8102290 2024-03-16T09:41:17.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.829+0000 7fc58ffff640 1 -- 172.21.15.77:0/4049476901 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fc5a009b750 con 0x7fc5a8102290 2024-03-16T09:41:17.832 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:41:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2890 msgr2=0x7fc5900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2890 0x7fc5900a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc598032380 tx=0x7fc598000f40 comp rx=0 tx=0).stop 2024-03-16T09:41:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8102290 msgr2=0x7fc5a819ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8102290 0x7fc5a819ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000e9b0 tx=0x7fc5a000ee70 comp rx=0 tx=0).stop 2024-03-16T09:41:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 shutdown_connections 2024-03-16T09:41:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc5900a2890 0x7fc5900a4d50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5a8102290 0x7fc5a819ee40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5a81018d0 0x7fc5a819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 --2- 172.21.15.77:0/4049476901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5a81006e0 0x7fc5a8069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 >> 172.21.15.77:0/4049476901 conn(0x7fc5a80fbc70 msgr2=0x7fc5a8104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 shutdown_connections 2024-03-16T09:41:17.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.831+0000 7fc5ad7f4640 1 -- 172.21.15.77:0/4049476901 wait complete. 2024-03-16T09:41:17.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 -- 172.21.15.77:0/327105312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 msgr2=0x7fdca4100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/327105312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4100e00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009900 tx=0x7fdc9802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 -- 172.21.15.77:0/327105312 shutdown_connections 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/327105312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca4101340 0x7fdca4108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/327105312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4100e00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/327105312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdca4101d30 0x7fdca4102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 -- 172.21.15.77:0/327105312 >> 172.21.15.77:0/327105312 conn(0x7fdca40fb410 msgr2=0x7fdca40fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:17.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 -- 172.21.15.77:0/327105312 shutdown_connections 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.946+0000 7fdcaab7e640 1 -- 172.21.15.77:0/327105312 wait complete. 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 Processor -- start 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 -- start start 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca4101340 0x7fdca4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdca4101d30 0x7fdca419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdca410d3d0 con 0x7fdca4100980 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdca410d250 con 0x7fdca4101d30 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdcaab7e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdca410d550 con 0x7fdca4101340 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52834/0 (socket says 172.21.15.77:52834) 2024-03-16T09:41:17.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 -- 172.21.15.77:0/2478746373 learned_addr learned my addr 172.21.15.77:0/2478746373 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca3fff640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca4101340 0x7fdca4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca911b640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdca4101d30 0x7fdca419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 -- 172.21.15.77:0/2478746373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca4101340 msgr2=0x7fdca4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca4101340 0x7fdca4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 -- 172.21.15.77:0/2478746373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdca4101d30 msgr2=0x7fdca419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdca4101d30 0x7fdca419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.947+0000 7fdca891a640 1 -- 172.21.15.77:0/2478746373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca419f690 con 0x7fdca4100980 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.948+0000 7fdca891a640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4197a70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400e9c0 tx=0x7fdc9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.948+0000 7fdca1ffb640 1 -- 172.21.15.77:0/2478746373 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9400cd80 con 0x7fdca4100980 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.948+0000 7fdca1ffb640 1 -- 172.21.15.77:0/2478746373 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc940035f0 con 0x7fdca4100980 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.948+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca419f980 con 0x7fdca4100980 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.948+0000 7fdca1ffb640 1 -- 172.21.15.77:0/2478746373 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc94010690 con 0x7fdca4100980 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.948+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca419fef0 con 0x7fdca4100980 2024-03-16T09:41:17.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.949+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca4068a20 con 0x7fdca4100980 2024-03-16T09:41:17.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.950+0000 7fdca1ffb640 1 -- 172.21.15.77:0/2478746373 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fdc940037a0 con 0x7fdca4100980 2024-03-16T09:41:17.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.950+0000 7fdca1ffb640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc8c0a2910 0x7fdc8c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:17.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.950+0000 7fdca3fff640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc8c0a2910 0x7fdc8c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:17.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.951+0000 7fdca1ffb640 1 -- 172.21.15.77:0/2478746373 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdc940ceef0 con 0x7fdca4100980 2024-03-16T09:41:17.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.951+0000 7fdca3fff640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc8c0a2910 0x7fdc8c0a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdc98005ab0 tx=0x7fdc98005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:17.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:17.955+0000 7fdca1ffb640 1 -- 172.21.15.77:0/2478746373 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc9409be40 con 0x7fdca4100980 2024-03-16T09:41:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.129+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdca402d1b0 con 0x7fdca4100980 2024-03-16T09:41:18.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.130+0000 7fdca1ffb640 1 -- 172.21.15.77:0/2478746373 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fdc9409b740 con 0x7fdca4100980 2024-03-16T09:41:18.133 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc8c0a2910 msgr2=0x7fdc8c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc8c0a2910 0x7fdc8c0a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdc98005ab0 tx=0x7fdc98005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 msgr2=0x7fdca4197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4197a70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400e9c0 tx=0x7fdc9400ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 shutdown_connections 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc8c0a2910 0x7fdc8c0a4dd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdca4101d30 0x7fdca419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca4101340 0x7fdca4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 --2- 172.21.15.77:0/2478746373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdca4100980 0x7fdca4197a70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 >> 172.21.15.77:0/2478746373 conn(0x7fdca40fb410 msgr2=0x7fdca4104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 shutdown_connections 2024-03-16T09:41:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.132+0000 7fdcaab7e640 1 -- 172.21.15.77:0/2478746373 wait complete. 2024-03-16T09:41:18.145 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:17 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2411699426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:18.145 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:17 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/134822666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:18.145 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:17 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2114091849' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:18.145 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:17 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4049476901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:18.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:41:18.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:17 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2411699426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:18.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:17 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/134822666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:18.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:17 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2114091849' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:18.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:17 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4049476901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 -- 172.21.15.77:0/781487672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c102810 msgr2=0x7fc50c102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 --2- 172.21.15.77:0/781487672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c102810 0x7fc50c102c90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009900 tx=0x7fc4f802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 -- 172.21.15.77:0/781487672 shutdown_connections 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 --2- 172.21.15.77:0/781487672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c1031d0 0x7fc50c109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 --2- 172.21.15.77:0/781487672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c102810 0x7fc50c102c90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 --2- 172.21.15.77:0/781487672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc50c101620 0x7fc50c101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 -- 172.21.15.77:0/781487672 >> 172.21.15.77:0/781487672 conn(0x7fc50c0fcb70 msgr2=0x7fc50c0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:18.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 -- 172.21.15.77:0/781487672 shutdown_connections 2024-03-16T09:41:18.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.244+0000 7fc511d9f640 1 -- 172.21.15.77:0/781487672 wait complete. 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 Processor -- start 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 -- start start 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc50c101620 0x7fc50c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c102810 0x7fc50c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c1031d0 0x7fc50c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c10d160 con 0x7fc50c1031d0 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c10cfe0 con 0x7fc50c102810 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc511d9f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c10d2e0 con 0x7fc50c101620 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c1031d0 0x7fc50c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c1031d0 0x7fc50c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52860/0 (socket says 172.21.15.77:52860) 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 -- 172.21.15.77:0/4136462565 learned_addr learned my addr 172.21.15.77:0/4136462565 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc50bfff640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c102810 0x7fc50c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc510d9d640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc50c101620 0x7fc50c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 -- 172.21.15.77:0/4136462565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc50c101620 msgr2=0x7fc50c1978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc50c101620 0x7fc50c1978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 -- 172.21.15.77:0/4136462565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c102810 msgr2=0x7fc50c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c102810 0x7fc50c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.245+0000 7fc51159e640 1 -- 172.21.15.77:0/4136462565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c19f4e0 con 0x7fc50c1031d0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.246+0000 7fc51159e640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c1031d0 0x7fc50c19ee70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc50400eb10 tx=0x7fc50400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.246+0000 7fc509ffb640 1 -- 172.21.15.77:0/4136462565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504004b20 con 0x7fc50c1031d0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.246+0000 7fc509ffb640 1 -- 172.21.15.77:0/4136462565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc504004cc0 con 0x7fc50c1031d0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.246+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50c19f7d0 con 0x7fc50c1031d0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.246+0000 7fc509ffb640 1 -- 172.21.15.77:0/4136462565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504010690 con 0x7fc50c1031d0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.246+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50c19fcf0 con 0x7fc50c1031d0 2024-03-16T09:41:18.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.247+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc50c1089e0 con 0x7fc50c1031d0 2024-03-16T09:41:18.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.248+0000 7fc509ffb640 1 -- 172.21.15.77:0/4136462565 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc504005020 con 0x7fc50c1031d0 2024-03-16T09:41:18.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.248+0000 7fc509ffb640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc4fc0a2910 0x7fc4fc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.248+0000 7fc510d9d640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc4fc0a2910 0x7fc4fc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.248+0000 7fc509ffb640 1 -- 172.21.15.77:0/4136462565 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fc5040cf0f0 con 0x7fc50c1031d0 2024-03-16T09:41:18.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.249+0000 7fc510d9d640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc4fc0a2910 0x7fc4fc0a4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc50c102670 tx=0x7fc500005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:18.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.253+0000 7fc509ffb640 1 -- 172.21.15.77:0/4136462565 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc50409c040 con 0x7fc50c1031d0 2024-03-16T09:41:18.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:17 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2411699426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:17 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/134822666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:17 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2114091849' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:17 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4049476901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:18.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.463+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc50c0623e0 con 0x7fc50c1031d0 2024-03-16T09:41:18.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.463+0000 7fc509ffb640 1 -- 172.21.15.77:0/4136462565 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11878 (secure 0 0 0) 0x7fc50409b940 con 0x7fc50c1031d0 2024-03-16T09:41:18.466 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:39:59.725431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24545,24503]},"id":1}]} 2024-03-16T09:41:18.467 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-03-16T09:41:18.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc4fc0a2910 msgr2=0x7fc4fc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc4fc0a2910 0x7fc4fc0a4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc50c102670 tx=0x7fc500005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c1031d0 msgr2=0x7fc50c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c1031d0 0x7fc50c19ee70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc50400eb10 tx=0x7fc50400c2d0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 shutdown_connections 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc4fc0a2910 0x7fc4fc0a4dd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc50c1031d0 0x7fc50c19ee70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c102810 0x7fc50c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 --2- 172.21.15.77:0/4136462565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc50c101620 0x7fc50c1978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.465+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 >> 172.21.15.77:0/4136462565 conn(0x7fc50c0fcb70 msgr2=0x7fc50c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.466+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 shutdown_connections 2024-03-16T09:41:18.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.466+0000 7fc511d9f640 1 -- 172.21.15.77:0/4136462565 wait complete. 2024-03-16T09:41:18.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:18.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 -- 172.21.15.77:0/130596876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac1022b0 msgr2=0x7f14ac108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 --2- 172.21.15.77:0/130596876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac1022b0 0x7f14ac108b40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009ca0 tx=0x7f14a402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 -- 172.21.15.77:0/130596876 shutdown_connections 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 --2- 172.21.15.77:0/130596876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac1022b0 0x7f14ac108b40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 --2- 172.21.15.77:0/130596876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14ac1018f0 0x7f14ac101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 --2- 172.21.15.77:0/130596876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14ac100700 0x7f14ac100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 -- 172.21.15.77:0/130596876 >> 172.21.15.77:0/130596876 conn(0x7f14ac0fbc90 msgr2=0x7f14ac0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 -- 172.21.15.77:0/130596876 shutdown_connections 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 -- 172.21.15.77:0/130596876 wait complete. 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 Processor -- start 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.577+0000 7f14b36b2640 1 -- start start 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac100700 0x7f14ac1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14ac1018f0 0x7f14ac197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14ac1022b0 0x7f14ac19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac10d350 con 0x7f14ac100700 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac10d1d0 con 0x7f14ac1022b0 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac10d4d0 con 0x7f14ac1018f0 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac100700 0x7f14ac1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac100700 0x7f14ac1979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52874/0 (socket says 172.21.15.77:52874) 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 -- 172.21.15.77:0/3891325154 learned_addr learned my addr 172.21.15.77:0/3891325154 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b0c4d640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14ac1018f0 0x7f14ac197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b1c4f640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14ac1022b0 0x7f14ac19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 -- 172.21.15.77:0/3891325154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14ac1018f0 msgr2=0x7f14ac197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14ac1018f0 0x7f14ac197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 -- 172.21.15.77:0/3891325154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14ac1022b0 msgr2=0x7f14ac19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14ac1022b0 0x7f14ac19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 -- 172.21.15.77:0/3891325154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ac19f5e0 con 0x7f14ac100700 2024-03-16T09:41:18.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b144e640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac100700 0x7f14ac1979c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f149c00e9c0 tx=0x7f149c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:18.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14a27fc640 1 -- 172.21.15.77:0/3891325154 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c00cd80 con 0x7f14ac100700 2024-03-16T09:41:18.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14a27fc640 1 -- 172.21.15.77:0/3891325154 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f149c0035f0 con 0x7f14ac100700 2024-03-16T09:41:18.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ac19f8d0 con 0x7f14ac100700 2024-03-16T09:41:18.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14a27fc640 1 -- 172.21.15.77:0/3891325154 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c010690 con 0x7f14ac100700 2024-03-16T09:41:18.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.578+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ac19fe20 con 0x7f14ac100700 2024-03-16T09:41:18.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.579+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ac107ac0 con 0x7f14ac100700 2024-03-16T09:41:18.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.580+0000 7f14a27fc640 1 -- 172.21.15.77:0/3891325154 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f149c0037a0 con 0x7f14ac100700 2024-03-16T09:41:18.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.581+0000 7f14a27fc640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f14980a2910 0x7f14980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.581+0000 7f14b0c4d640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f14980a2910 0x7f14980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.581+0000 7f14a27fc640 1 -- 172.21.15.77:0/3891325154 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f149c0cf120 con 0x7f14ac100700 2024-03-16T09:41:18.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.581+0000 7f14b0c4d640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f14980a2910 0x7f14980a4dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f14ac198ed0 tx=0x7f14a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:18.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.585+0000 7f14a27fc640 1 -- 172.21.15.77:0/3891325154 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f149c09c010 con 0x7f14ac100700 2024-03-16T09:41:18.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.762+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14ac0623e0 con 0x7f14ac100700 2024-03-16T09:41:18.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.763+0000 7f14a27fc640 1 -- 172.21.15.77:0/3891325154 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f149c09b910 con 0x7f14ac100700 2024-03-16T09:41:18.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:18.766 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:18.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f14980a2910 msgr2=0x7f14980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f14980a2910 0x7f14980a4dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f14ac198ed0 tx=0x7f14a8005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:18.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac100700 msgr2=0x7f14ac1979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac100700 0x7f14ac1979c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f149c00e9c0 tx=0x7f149c00ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:18.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 shutdown_connections 2024-03-16T09:41:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f14980a2910 0x7f14980a4dd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14ac1022b0 0x7f14ac19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14ac1018f0 0x7f14ac197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 --2- 172.21.15.77:0/3891325154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14ac100700 0x7f14ac1979c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 >> 172.21.15.77:0/3891325154 conn(0x7f14ac0fbc90 msgr2=0x7f14ac104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 shutdown_connections 2024-03-16T09:41:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.765+0000 7f14b36b2640 1 -- 172.21.15.77:0/3891325154 wait complete. 2024-03-16T09:41:18.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.879+0000 7f5268537640 1 -- 172.21.15.77:0/3488164394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 msgr2=0x7f5260101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.879+0000 7f5268537640 1 --2- 172.21.15.77:0/3488164394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260101d70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5250009900 tx=0x7f525002bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- 172.21.15.77:0/3488164394 shutdown_connections 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 --2- 172.21.15.77:0/3488164394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52601022b0 0x7f5260108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 --2- 172.21.15.77:0/3488164394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260101d70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 --2- 172.21.15.77:0/3488164394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5260100700 0x7f5260100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- 172.21.15.77:0/3488164394 >> 172.21.15.77:0/3488164394 conn(0x7f52600fbc90 msgr2=0x7f52600fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- 172.21.15.77:0/3488164394 shutdown_connections 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- 172.21.15.77:0/3488164394 wait complete. 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 Processor -- start 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- start start 2024-03-16T09:41:18.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5260100700 0x7f52601048d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52601022b0 0x7f5260103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f526010c230 con 0x7f52601018f0 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f526010c0b0 con 0x7f52601022b0 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5268537640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f526010c3b0 con 0x7f5260100700 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5265ad2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5265ad2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260102f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52894/0 (socket says 172.21.15.77:52894) 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5265ad2640 1 -- 172.21.15.77:0/1664711165 learned_addr learned my addr 172.21.15.77:0/1664711165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f52662d3640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5260100700 0x7f52601048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.880+0000 7f5266ad4640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52601022b0 0x7f5260103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5265ad2640 1 -- 172.21.15.77:0/1664711165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5260100700 msgr2=0x7f52601048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5265ad2640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5260100700 0x7f52601048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5265ad2640 1 -- 172.21.15.77:0/1664711165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52601022b0 msgr2=0x7f5260103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5265ad2640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52601022b0 0x7f5260103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5265ad2640 1 -- 172.21.15.77:0/1664711165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5260103ce0 con 0x7f52601018f0 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5265ad2640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260102f10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5250006010 tx=0x7f5250005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f52577fe640 1 -- 172.21.15.77:0/1664711165 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250043020 con 0x7f52601018f0 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52601a1b70 con 0x7f52601018f0 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f52577fe640 1 -- 172.21.15.77:0/1664711165 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52500043d0 con 0x7f52601018f0 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f52577fe640 1 -- 172.21.15.77:0/1664711165 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250003a80 con 0x7f52601018f0 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.881+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52601a20c0 con 0x7f52601018f0 2024-03-16T09:41:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.882+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52601a2290 con 0x7f52601018f0 2024-03-16T09:41:18.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.883+0000 7f52577fe640 1 -- 172.21.15.77:0/1664711165 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5250046020 con 0x7f52601018f0 2024-03-16T09:41:18.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.883+0000 7f52577fe640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f524c0a2890 0x7f524c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:18.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.884+0000 7f52662d3640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f524c0a2890 0x7f524c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:18.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.884+0000 7f52577fe640 1 -- 172.21.15.77:0/1664711165 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f52500f12a0 con 0x7f52601018f0 2024-03-16T09:41:18.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.884+0000 7f52662d3640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f524c0a2890 0x7f524c0a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5260101750 tx=0x7f5258009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:18.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:18.887+0000 7f52577fe640 1 -- 172.21.15.77:0/1664711165 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f525004d020 con 0x7f52601018f0 2024-03-16T09:41:19.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.100+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52601a28e0 con 0x7f52601018f0 2024-03-16T09:41:19.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.101+0000 7f52577fe640 1 -- 172.21.15.77:0/1664711165 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11878 (secure 0 0 0) 0x7f52500be300 con 0x7f52601018f0 2024-03-16T09:41:19.104 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:19.104 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:39:59.725431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24545,24503]},"id":1}]} 2024-03-16T09:41:19.104 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-03-16T09:41:19.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f524c0a2890 msgr2=0x7f524c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f524c0a2890 0x7f524c0a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5260101750 tx=0x7f5258009500 comp rx=0 tx=0).stop 2024-03-16T09:41:19.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 msgr2=0x7f5260102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260102f10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5250006010 tx=0x7f5250005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:19.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 shutdown_connections 2024-03-16T09:41:19.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f524c0a2890 0x7f524c0a4d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52601022b0 0x7f5260103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52601018f0 0x7f5260102f10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 --2- 172.21.15.77:0/1664711165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5260100700 0x7f52601048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 >> 172.21.15.77:0/1664711165 conn(0x7f52600fbc90 msgr2=0x7f5260069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 shutdown_connections 2024-03-16T09:41:19.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.103+0000 7f5268537640 1 -- 172.21.15.77:0/1664711165 wait complete. 2024-03-16T09:41:19.115 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:18 smithi077 ceph-mon[32171]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:19.116 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:18 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2478746373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:19.116 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:18 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4136462565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:19.116 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:18 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3891325154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:19.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T09:39:55.509822+0000', 'modified': '2024-03-16T09:39:59.725431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24545, 'mds_1': 24503, 'mds_2': 24481}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6835/2328435031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 2328435031}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 2328435031}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6837/814033341', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 814033341}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 814033341}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/296042620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 296042620}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 296042620}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6839/1514269525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 1514269525}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 1514269525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.119:6837/1948234265', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 1948234265}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 1948234265}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/3029827381', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 3029827381}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 3029827381}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24545, 'qdb_cluster': [24481, 24545, 24503]} 2024-03-16T09:41:19.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T09:41:19.117 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T09:41:19.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:18 smithi119 ceph-mon[35632]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:19.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:18 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2478746373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:19.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:18 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4136462565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:19.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:18 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3891325154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.216+0000 7f8bc3737640 1 -- 172.21.15.77:0/3598292500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 msgr2=0x7f8bbc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.216+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3598292500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc101c10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009900 tx=0x7f8bb802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- 172.21.15.77:0/3598292500 shutdown_connections 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3598292500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bbc1033a0 0x7f8bbc109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3598292500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bbc1029e0 0x7f8bbc102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3598292500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc101c10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- 172.21.15.77:0/3598292500 >> 172.21.15.77:0/3598292500 conn(0x7f8bbc0fcd80 msgr2=0x7f8bbc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- 172.21.15.77:0/3598292500 shutdown_connections 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- 172.21.15.77:0/3598292500 wait complete. 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 Processor -- start 2024-03-16T09:41:19.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- start start 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bbc1029e0 0x7f8bbc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bbc1033a0 0x7f8bbc19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc10d310 con 0x7f8bbc1017f0 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc10d190 con 0x7f8bbc1033a0 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc3737640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc10d490 con 0x7f8bbc1029e0 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc14d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc14d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52920/0 (socket says 172.21.15.77:52920) 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.217+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3791357275 learned_addr learned my addr 172.21.15.77:0/3791357275 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bbc1033a0 0x7f8bbc19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bbc1029e0 0x7f8bbc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3791357275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bbc1029e0 msgr2=0x7f8bbc197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bbc1029e0 0x7f8bbc197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3791357275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bbc1033a0 msgr2=0x7f8bbc19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bbc1033a0 0x7f8bbc19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3791357275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc19f6d0 con 0x7f8bbc1017f0 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bbc1029e0 0x7f8bbc197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc197ab0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000c00 tx=0x7f8bb8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bb27fc640 1 -- 172.21.15.77:0/3791357275 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8043020 con 0x7f8bbc1017f0 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bb27fc640 1 -- 172.21.15.77:0/3791357275 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bb8004350 con 0x7f8bbc1017f0 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc19f960 con 0x7f8bbc1017f0 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bb27fc640 1 -- 172.21.15.77:0/3791357275 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8003b10 con 0x7f8bbc1017f0 2024-03-16T09:41:19.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.218+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc19ff00 con 0x7f8bbc1017f0 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.220+0000 7f8bb27fc640 1 -- 172.21.15.77:0/3791357275 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8bb8046020 con 0x7f8bbc1017f0 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 Processor -- start 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 -- start start 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b8aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b900c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b92bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c174aeec80 con 0x55c174b8abd0 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c174a97aa0 con 0x55c174b90730 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c174aead20 con 0x55c174b8b5c0 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bb27fc640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8ba80a2910 0x7f8ba80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b8aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc14d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b900c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b8aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52922/0 (socket says 172.21.15.77:52922) 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1869120282 learned_addr learned my addr 172.21.15.77:0/1869120282 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b92bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8ba80a2910 0x7f8ba80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bb27fc640 1 -- 172.21.15.77:0/3791357275 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8bb80f0c70 con 0x7f8bbc1017f0 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1869120282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 msgr2=0x55c174b900c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b900c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1869120282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 msgr2=0x55c174b92bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b92bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1869120282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c174ab9ff0 con 0x55c174b8abd0 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.221+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b92bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b8aff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4011e20 tx=0x7f8bb400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845038a64c2fa53f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8ba80a2910 0x7f8ba80a4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400fc90 tx=0x7f8bb401e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8ba77fe640 1 -- 172.21.15.77:0/1869120282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb402a070 con 0x55c174b8abd0 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8ba77fe640 1 -- 172.21.15.77:0/1869120282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bb4009e40 con 0x55c174b8abd0 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8ba77fe640 1 -- 172.21.15.77:0/1869120282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4004490 con 0x55c174b8abd0 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1869120282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 msgr2=0x55c174b8aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b8aff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4011e20 tx=0x7f8bb400f790 comp rx=0 tx=0).stop 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1869120282 shutdown_connections 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b92bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b900c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1869120282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b8aff0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.222+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1869120282 >> 172.21.15.77:0/1869120282 conn(0x55c174b86140 msgr2=0x55c174b86540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1869120282 shutdown_connections 2024-03-16T09:41:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1869120282 wait complete. 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 -- start start 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 Processor -- start 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8abd0 0x55c174b94d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b952c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174b9c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c174b93680 con 0x55c174b90730 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c174aaaef0 con 0x55c174b8abd0 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c174b93800 con 0x55c174b8b5c0 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc63f8740 10 client.0 _ll_register_callbacks cb 0x55c174aa5dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc1cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174b9c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc1cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174b9c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52938/0 (socket says 172.21.15.77:52938) 2024-03-16T09:41:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc14d3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8abd0 0x55c174b94d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/2938293286 learned_addr learned my addr 172.21.15.77:0/2938293286 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.223+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 msgr2=0x55c174b952c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b952c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/2938293286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8abd0 msgr2=0x55c174b94d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8abd0 0x55c174b94d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/2938293286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c174ab7a30 con 0x55c174b90730 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8abd0 0x55c174b94d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174b9c300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4005c10 tx=0x7f8bb4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9de3db453d4b40e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8ba4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/2938293286 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb402a040 con 0x55c174b90730 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/2938293286 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8bb4003a70 con 0x55c174b90730 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c174afc1c0 con 0x55c174b90730 2024-03-16T09:41:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/2938293286 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f8bb4003c70 con 0x55c174b90730 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc63f8740 10 client.15039 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c1748032d0 con 0x55c174b90730 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/2938293286 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c174afc1c0 con 0x55c174b90730 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/2938293286 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f8bb403b4a0 con 0x55c174b90730 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.224+0000 7f8bc63f8740 10 client.15039 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8bc63f8740 10 client.15039 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8bc63f8740 20 client.15039 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174ba1640 0x55c174ba2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8bc63f8740 4 client.15039 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c174af6510 con 0x55c174ba1640 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174ba1640 0x55c174ba2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8ba4ff9640 10 client.15039 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.225+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174ba1640 0x55c174ba2ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e0aca3f949f5219 server_cookie=9e83fba9dbf12b48 in_seq=0 out_seq=0 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/2938293286 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c174af6510 con 0x55c174ba1640 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8ba4ff9640 10 client.15039 handle_command_reply: tid=0 2024-03-16T09:41:19.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 client.15039 shutdown 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 msgr2=0x55c174b9c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174b9c300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4005c10 tx=0x7f8bb4011a50 comp rx=0 tx=0).stop 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 shutdown_connections 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174b9c300 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b952c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174ba1640 0x55c174ba2ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/2938293286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8abd0 0x55c174b94d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8ba4ff9640 0 client.15039 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 >> 172.21.15.77:0/2938293286 conn(0x55c174b86140 msgr2=0x55c174b85380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 shutdown_connections 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 1 -- 172.21.15.77:0/2938293286 wait complete. 2024-03-16T09:41:19.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.226+0000 7f8bc63f8740 20 client.15039 trim_cache size 0 max 16384 2024-03-16T09:41:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 Processor -- start 2024-03-16T09:41:19.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 -- start start 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b86520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b84af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b85030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c174ba0df0 con 0x55c174b8abd0 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c174ba14c0 con 0x55c174b90730 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c174ba1b90 con 0x55c174b8b5c0 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc14d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b86520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc14d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b86520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52950/0 (socket says 172.21.15.77:52950) 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3827450796 learned_addr learned my addr 172.21.15.77:0/3827450796 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b84af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b85030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3827450796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 msgr2=0x55c174b84af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b84af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3827450796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 msgr2=0x55c174b85030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.244+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b85030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc14d3640 1 -- 172.21.15.77:0/3827450796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1748cd6b0 con 0x55c174b8abd0 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b84af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b86520 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80b9a30 tx=0x7f8bb80a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95709d9dd36d671 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8ba6ffd640 1 -- 172.21.15.77:0/3827450796 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80bdcd0 con 0x55c174b8abd0 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8ba6ffd640 1 -- 172.21.15.77:0/3827450796 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bb80bd5d0 con 0x55c174b8abd0 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8ba6ffd640 1 -- 172.21.15.77:0/3827450796 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8003cb0 con 0x55c174b8abd0 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 -- 172.21.15.77:0/3827450796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 msgr2=0x55c174b86520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b86520 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80b9a30 tx=0x7f8bb80a8040 comp rx=0 tx=0).stop 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 -- 172.21.15.77:0/3827450796 shutdown_connections 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b90730 0x55c174b85030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8b5c0 0x55c174b84af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/3827450796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b8abd0 0x55c174b86520 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 -- 172.21.15.77:0/3827450796 >> 172.21.15.77:0/3827450796 conn(0x55c174b86140 msgr2=0x55c174ba05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 -- 172.21.15.77:0/3827450796 shutdown_connections 2024-03-16T09:41:19.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.245+0000 7f8bc63f8740 1 -- 172.21.15.77:0/3827450796 wait complete. 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 -- start start 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 Processor -- start 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8abd0 0x55c174b9b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8b5c0 0x55c174b9b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174c094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c174ba2930 con 0x55c174b90730 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c174ba2260 con 0x55c174b8b5c0 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c174ba3000 con 0x55c174b8abd0 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc63f8740 10 client.0 _ll_register_callbacks cb 0x7f8bb4028070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174c094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174c094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52954/0 (socket says 172.21.15.77:52954) 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1209326554 learned_addr learned my addr 172.21.15.77:0/1209326554 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc0cd2640 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8b5c0 0x55c174b9b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8abd0 0x55c174b9b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8abd0 msgr2=0x55c174b9b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8abd0 0x55c174b9b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1209326554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8b5c0 msgr2=0x55c174b9b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8b5c0 0x55c174b9b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.246+0000 7f8bc1cd4640 1 -- 172.21.15.77:0/1209326554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c174c09ae0 con 0x55c174b90730 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc1cd4640 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174c094a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8bb403b8e0 tx=0x7f8bb4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bc26be783a80de server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8ba4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/1209326554 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb402a040 con 0x55c174b90730 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/1209326554 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8bb4044480 con 0x55c174b90730 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bb80bd5d0 con 0x55c174b90730 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/1209326554 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f8bb4009500 con 0x55c174b90730 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 10 client.15045 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:19.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c174c0ae20 con 0x55c174b90730 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/1209326554 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bb80bd5d0 con 0x55c174b90730 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/1209326554 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f8bb4005070 con 0x55c174b90730 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 10 client.15045 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 10 client.15045 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 20 client.15045 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174c0fa30 0x55c174c0e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 4 client.15045 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "status"}) -- 0x55c174a74a00 con 0x55c174c0fa30 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.247+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174c0fa30 0x55c174c0e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8ba4ff9640 10 client.15045 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc14d3640 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174c0fa30 0x55c174c0e740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f93849e9d81bed6a server_cookie=fbc5db8d068e7226 in_seq=0 out_seq=0 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8ba4ff9640 1 -- 172.21.15.77:0/1209326554 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c174a74a00 con 0x55c174c0fa30 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8ba4ff9640 10 client.15045 handle_command_reply: tid=0 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 client.15045 shutdown 2024-03-16T09:41:19.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 msgr2=0x55c174c094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174c094a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8bb403b8e0 tx=0x7f8bb4003ba0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 shutdown_connections 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c174b90730 0x55c174c094a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55c174c0fa30 0x55c174c0e740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c174b8b5c0 0x55c174b9b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8ba4ff9640 0 client.15045 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 --2- 172.21.15.77:0/1209326554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c174b8abd0 0x55c174b9b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 >> 172.21.15.77:0/1209326554 conn(0x55c174b86140 msgr2=0x55c174b86520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 shutdown_connections 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 1 -- 172.21.15.77:0/1209326554 wait complete. 2024-03-16T09:41:19.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.248+0000 7f8bc63f8740 20 client.15045 trim_cache size 0 max 16384 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 2, 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24481, 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 80.525766141000005, 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 20, 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 88, 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 86, 2024-03-16T09:41:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 96.914608745999999 2024-03-16T09:41:19.260 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:19.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8ba80a2910 msgr2=0x7f8ba80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8ba80a2910 0x7f8ba80a4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400fc90 tx=0x7f8bb401e000 comp rx=0 tx=0).stop 2024-03-16T09:41:19.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 msgr2=0x7f8bbc197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc197ab0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8000c00 tx=0x7f8bb8005dc0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 shutdown_connections 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8ba80a2910 0x7f8ba80a4dd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bbc1033a0 0x7f8bbc19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bbc1029e0 0x7f8bbc197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 --2- 172.21.15.77:0/3791357275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bbc1017f0 0x7f8bbc197ab0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 >> 172.21.15.77:0/3791357275 conn(0x7f8bbc0fcd80 msgr2=0x7f8bbc105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 shutdown_connections 2024-03-16T09:41:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.249+0000 7f8bc3737640 1 -- 172.21.15.77:0/3791357275 wait complete. 2024-03-16T09:41:19.262 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T09:41:19.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:18 smithi129 ceph-mon[35841]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:18 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2478746373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:18 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4136462565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:18 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3891325154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 -- 172.21.15.77:0/273086644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 msgr2=0x7f1c68109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 --2- 172.21.15.77:0/273086644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c68109c30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009ca0 tx=0x7f1c6002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 -- 172.21.15.77:0/273086644 shutdown_connections 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 --2- 172.21.15.77:0/273086644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c68109c30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 --2- 172.21.15.77:0/273086644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c681029e0 0x7f1c68102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 --2- 172.21.15.77:0/273086644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c681017f0 0x7f1c68101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 -- 172.21.15.77:0/273086644 >> 172.21.15.77:0/273086644 conn(0x7f1c680fcd80 msgr2=0x7f1c680ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 -- 172.21.15.77:0/273086644 shutdown_connections 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.361+0000 7f1c6e861640 1 -- 172.21.15.77:0/273086644 wait complete. 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 Processor -- start 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 -- start start 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c681017f0 0x7f1c681979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c681029e0 0x7f1c68197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c6819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6810d300 con 0x7f1c681033a0 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6810d180 con 0x7f1c681029e0 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6e861640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6810d480 con 0x7f1c681017f0 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c6819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c6819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52964/0 (socket says 172.21.15.77:52964) 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 -- 172.21.15.77:0/2395667684 learned_addr learned my addr 172.21.15.77:0/2395667684 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c681017f0 0x7f1c681979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c681029e0 0x7f1c68197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 -- 172.21.15.77:0/2395667684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c681017f0 msgr2=0x7f1c681979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c681017f0 0x7f1c681979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 -- 172.21.15.77:0/2395667684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c681029e0 msgr2=0x7f1c68197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c681029e0 0x7f1c68197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.362+0000 7f1c6cdfe640 1 -- 172.21.15.77:0/2395667684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6819f5d0 con 0x7f1c681033a0 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.363+0000 7f1c6cdfe640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c6819ef60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1c6002c700 tx=0x7f1c600328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.363+0000 7f1c657fa640 1 -- 172.21.15.77:0/2395667684 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60041020 con 0x7f1c681033a0 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.363+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6819f860 con 0x7f1c681033a0 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.363+0000 7f1c657fa640 1 -- 172.21.15.77:0/2395667684 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c60003e20 con 0x7f1c681033a0 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.363+0000 7f1c657fa640 1 -- 172.21.15.77:0/2395667684 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60039040 con 0x7f1c681033a0 2024-03-16T09:41:19.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.363+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6819fda0 con 0x7f1c681033a0 2024-03-16T09:41:19.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c657fa640 1 -- 172.21.15.77:0/2395667684 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1c60044020 con 0x7f1c681033a0 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 Processor -- start 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 -- start start 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d210c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d23bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd86c7fc80 con 0x55bd86d1c5c0 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd86c28aa0 con 0x55bd86d21730 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c71451740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bd86c7bd20 con 0x55bd86d1bbd0 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d210c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d210c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52980/0 (socket says 172.21.15.77:52980) 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c67fff640 1 -- 172.21.15.77:0/2522373604 learned_addr learned my addr 172.21.15.77:0/2522373604 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.365+0000 7f1c677fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d23bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c657fa640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c500a2890 0x7f1c500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c67fff640 1 -- 172.21.15.77:0/2522373604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1bbd0 msgr2=0x55bd86d1bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2522373604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c67fff640 1 -- 172.21.15.77:0/2522373604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 msgr2=0x55bd86d23bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2522373604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d23bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c6cdfe640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c500a2890 0x7f1c500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c67fff640 1 -- 172.21.15.77:0/2522373604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd86c4aff0 con 0x55bd86d1c5c0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c657fa640 1 -- 172.21.15.77:0/2395667684 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1c600f0c60 con 0x7f1c681033a0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2522373604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d210c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009ca0 tx=0x7f1c58011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88a00a3f63ae5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c4e7fc640 1 -- 172.21.15.77:0/2522373604 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58004220 con 0x55bd86d1c5c0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c4e7fc640 1 -- 172.21.15.77:0/2522373604 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c580043c0 con 0x55bd86d1c5c0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c4e7fc640 1 -- 172.21.15.77:0/2522373604 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58019810 con 0x55bd86d1c5c0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.366+0000 7f1c6cdfe640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c500a2890 0x7f1c500a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c600bd550 tx=0x7f1c6008f240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 -- 172.21.15.77:0/2522373604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 msgr2=0x55bd86d210c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 --2- 172.21.15.77:0/2522373604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d210c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009ca0 tx=0x7f1c58011810 comp rx=0 tx=0).stop 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 -- 172.21.15.77:0/2522373604 shutdown_connections 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 --2- 172.21.15.77:0/2522373604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d23bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 --2- 172.21.15.77:0/2522373604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d210c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 --2- 172.21.15.77:0/2522373604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 -- 172.21.15.77:0/2522373604 >> 172.21.15.77:0/2522373604 conn(0x55bd86d17140 msgr2=0x55bd86d17540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 -- 172.21.15.77:0/2522373604 shutdown_connections 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 -- 172.21.15.77:0/2522373604 wait complete. 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 -- start start 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 Processor -- start 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d25de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d26320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d2d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.367+0000 7f1c71451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d246e0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c71451740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd86c3bef0 con 0x55bd86d21730 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c71451740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d24860 con 0x55bd86d1c5c0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c71451740 10 client.0 _ll_register_callbacks cb 0x55bd86c36dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d25de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d25de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52984/0 (socket says 172.21.15.77:52984) 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 -- 172.21.15.77:0/2653122658 learned_addr learned my addr 172.21.15.77:0/2653122658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d26320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d2d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 -- 172.21.15.77:0/2653122658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 msgr2=0x55bd86d26320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d26320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 -- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 msgr2=0x55bd86d2d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d2d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 -- 172.21.15.77:0/2653122658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd86c48a30 con 0x55bd86d1bbd0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d25de0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0058c0 tx=0x7f1c5c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28755467bc1c0937 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c477fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c477fe640 1 -- 172.21.15.77:0/2653122658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c012270 con 0x55bd86d1bbd0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c477fe640 1 -- 172.21.15.77:0/2653122658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1c5c004030 con 0x55bd86d1bbd0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c580043c0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c477fe640 1 -- 172.21.15.77:0/2653122658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1c5c028050 con 0x55bd86d1bbd0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c71451740 10 client.15057 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.368+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bd869942d0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c477fe640 1 -- 172.21.15.77:0/2653122658 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c580043c0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c477fe640 1 -- 172.21.15.77:0/2653122658 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f1c5c003c30 con 0x55bd86d1bbd0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c71451740 10 client.15057 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c71451740 10 client.15057 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c71451740 20 client.15057 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c71451740 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86d327b0 0x55bd86d33c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c71451740 4 client.15057 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd86c8d1c0 con 0x55bd86d327b0 2024-03-16T09:41:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86d327b0 0x55bd86d33c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c477fe640 10 client.15057 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.369+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86d327b0 0x55bd86d33c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9a19623188cbed7 server_cookie=bbe96f34c5651243 in_seq=0 out_seq=0 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c477fe640 1 -- 172.21.15.77:0/2653122658 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd86c8d1c0 con 0x55bd86d327b0 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c477fe640 10 client.15057 handle_command_reply: tid=0 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 client.15057 shutdown 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 msgr2=0x55bd86d25de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d25de0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0058c0 tx=0x7f1c5c005850 comp rx=0 tx=0).stop 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 shutdown_connections 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86d327b0 0x55bd86d33c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d2d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c477fe640 0 client.15057 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d26320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 --2- 172.21.15.77:0/2653122658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d25de0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 >> 172.21.15.77:0/2653122658 conn(0x55bd86d17140 msgr2=0x55bd86d16380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 shutdown_connections 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 1 -- 172.21.15.77:0/2653122658 wait complete. 2024-03-16T09:41:19.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.370+0000 7f1c71451740 20 client.15057 trim_cache size 0 max 16384 2024-03-16T09:41:19.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 Processor -- start 2024-03-16T09:41:19.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 -- start start 2024-03-16T09:41:19.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d1bbd0 0x55bd86d259f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d25f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d21730 0x55bd86d15af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d31f60 con 0x55bd86d1c5c0 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d32630 con 0x55bd86d1bbd0 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c71451740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d32d00 con 0x55bd86d21730 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d25f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d25f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52992/0 (socket says 172.21.15.77:52992) 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 -- 172.21.15.77:0/1093873213 learned_addr learned my addr 172.21.15.77:0/1093873213 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c677fe640 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d1bbd0 0x55bd86d259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c677fe640 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d21730 0x55bd86d15af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 -- 172.21.15.77:0/1093873213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d21730 msgr2=0x55bd86d15af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d21730 0x55bd86d15af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 -- 172.21.15.77:0/1093873213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d1bbd0 msgr2=0x55bd86d259f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d1bbd0 0x55bd86d259f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c67fff640 1 -- 172.21.15.77:0/1093873213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd86a5e6b0 con 0x55bd86d1c5c0 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.388+0000 7f1c677fe640 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d1bbd0 0x55bd86d259f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c67fff640 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d25f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009380 tx=0x7f1c58009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b25fa40282515a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c4dffb640 1 -- 172.21.15.77:0/1093873213 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58032070 con 0x55bd86d1c5c0 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c4dffb640 1 -- 172.21.15.77:0/1093873213 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c58003aa0 con 0x55bd86d1c5c0 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c4dffb640 1 -- 172.21.15.77:0/1093873213 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c580052a0 con 0x55bd86d1c5c0 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 -- 172.21.15.77:0/1093873213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 msgr2=0x55bd86d25f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d25f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009380 tx=0x7f1c58009310 comp rx=0 tx=0).stop 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 -- 172.21.15.77:0/1093873213 shutdown_connections 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d21730 0x55bd86d15af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1c5c0 0x55bd86d25f30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 --2- 172.21.15.77:0/1093873213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d1bbd0 0x55bd86d259f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 -- 172.21.15.77:0/1093873213 >> 172.21.15.77:0/1093873213 conn(0x55bd86d17140 msgr2=0x55bd86d20de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 -- 172.21.15.77:0/1093873213 shutdown_connections 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 -- 172.21.15.77:0/1093873213 wait complete. 2024-03-16T09:41:19.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.389+0000 7f1c71451740 1 -- start start 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 1 Processor -- start 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d1fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d9ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d33aa0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d333d0 con 0x55bd86d21730 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bd86d34170 con 0x55bd86d1c5c0 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c71451740 10 client.0 _ll_register_callbacks cb 0x7f1c5c009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53000/0 (socket says 172.21.15.77:53000) 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 -- 172.21.15.77:0/2181304973 learned_addr learned my addr 172.21.15.77:0/2181304973 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d1fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d9ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 -- 172.21.15.77:0/2181304973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 msgr2=0x55bd86d1fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d1fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 -- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 msgr2=0x55bd86d9ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d9ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 -- 172.21.15.77:0/2181304973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd86d9b170 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c677fe640 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1f880 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c010480 tx=0x7f1c5c0124d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d9c3ba4a1ed2c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.390+0000 7f1c477fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c477fe640 1 -- 172.21.15.77:0/2181304973 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0042e0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c477fe640 1 -- 172.21.15.77:0/2181304973 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1c5c004480 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c58003aa0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c477fe640 1 -- 172.21.15.77:0/2181304973 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1c5c005040 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 10 client.15063 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bd86d9c4e0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c477fe640 1 -- 172.21.15.77:0/2181304973 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c58003aa0 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c477fe640 1 -- 172.21.15.77:0/2181304973 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f1c5c004b40 con 0x55bd86d1bbd0 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 10 client.15063 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:19.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 10 client.15063 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 20 client.15063 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86da1130 0x55bd86d9d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 4 client.15063 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "status"}) -- 0x55bd86c87510 con 0x55bd86da1130 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.391+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86da1130 0x55bd86d9d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c477fe640 10 client.15063 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c67fff640 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86da1130 0x55bd86d9d570 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e42c7d26b8edb0a server_cookie=cf554c3394a9c68c in_seq=0 out_seq=0 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c477fe640 1 -- 172.21.15.77:0/2181304973 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55bd86c87510 con 0x55bd86da1130 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c477fe640 10 client.15063 handle_command_reply: tid=0 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 client.15063 shutdown 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 msgr2=0x55bd86d1f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1f880 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c010480 tx=0x7f1c5c0124d0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 shutdown_connections 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55bd86da1130 0x55bd86d9d570 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd86d21730 0x55bd86d9ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c477fe640 0 client.15063 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:19.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd86d1c5c0 0x55bd86d1fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 --2- 172.21.15.77:0/2181304973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd86d1bbd0 0x55bd86d1f880 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 >> 172.21.15.77:0/2181304973 conn(0x55bd86d17140 msgr2=0x55bd86d259f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 shutdown_connections 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 1 -- 172.21.15.77:0/2181304973 wait complete. 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.392+0000 7f1c71451740 20 client.15063 trim_cache size 0 max 16384 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 1, 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24503, 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T09:41:19.402 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T09:41:19.403 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T09:41:19.403 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 81.68939297, 2024-03-16T09:41:19.403 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 20, 2024-03-16T09:41:19.403 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 88, 2024-03-16T09:41:19.403 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 86, 2024-03-16T09:41:19.403 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 99.693118032000001 2024-03-16T09:41:19.403 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c500a2890 msgr2=0x7f1c500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c500a2890 0x7f1c500a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c600bd550 tx=0x7f1c6008f240 comp rx=0 tx=0).stop 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 msgr2=0x7f1c6819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c6819ef60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1c6002c700 tx=0x7f1c600328a0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 shutdown_connections 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1c500a2890 0x7f1c500a4d50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c681033a0 0x7f1c6819ef60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c681029e0 0x7f1c68197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 --2- 172.21.15.77:0/2395667684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c681017f0 0x7f1c681979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 >> 172.21.15.77:0/2395667684 conn(0x7f1c680fcd80 msgr2=0x7f1c68105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 shutdown_connections 2024-03-16T09:41:19.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.393+0000 7f1c6e861640 1 -- 172.21.15.77:0/2395667684 wait complete. 2024-03-16T09:41:19.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T09:41:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 -- 172.21.15.77:0/2320212529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8102300 msgr2=0x7f05a81046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 --2- 172.21.15.77:0/2320212529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8102300 0x7f05a81046f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009900 tx=0x7f05a402bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 -- 172.21.15.77:0/2320212529 shutdown_connections 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 --2- 172.21.15.77:0/2320212529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05a8107560 0x7f05a8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 --2- 172.21.15.77:0/2320212529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8104c30 0x7f05a8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 --2- 172.21.15.77:0/2320212529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8102300 0x7f05a81046f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 -- 172.21.15.77:0/2320212529 >> 172.21.15.77:0/2320212529 conn(0x7f05a80fbc70 msgr2=0x7f05a80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 -- 172.21.15.77:0/2320212529 shutdown_connections 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.506+0000 7f05b0697640 1 -- 172.21.15.77:0/2320212529 wait complete. 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 Processor -- start 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 -- start start 2024-03-16T09:41:19.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8102300 0x7f05a8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8104c30 0x7f05a8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05a8107560 0x7f05a819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05a810d330 con 0x7f05a8104c30 2024-03-16T09:41:19.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f05a810d1b0 con 0x7f05a8107560 2024-03-16T09:41:19.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f05a810d4b0 con 0x7f05a8102300 2024-03-16T09:41:19.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8104c30 0x7f05a8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8104c30 0x7f05a8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53014/0 (socket says 172.21.15.77:53014) 2024-03-16T09:41:19.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 -- 172.21.15.77:0/2581099356 learned_addr learned my addr 172.21.15.77:0/2581099356 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05ae433640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8102300 0x7f05a8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05aec34640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05a8107560 0x7f05a819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 -- 172.21.15.77:0/2581099356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8102300 msgr2=0x7f05a8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8102300 0x7f05a8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 -- 172.21.15.77:0/2581099356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05a8107560 msgr2=0x7f05a819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05a8107560 0x7f05a819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 -- 172.21.15.77:0/2581099356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a819f6e0 con 0x7f05a8104c30 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05adc32640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8104c30 0x7f05a8198000 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f059800e9f0 tx=0x7f059800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f059f7fe640 1 -- 172.21.15.77:0/2581099356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059800ccc0 con 0x7f05a8104c30 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a819f9d0 con 0x7f05a8104c30 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f059f7fe640 1 -- 172.21.15.77:0/2581099356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05980054d0 con 0x7f05a8104c30 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.507+0000 7f059f7fe640 1 -- 172.21.15.77:0/2581099356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598010430 con 0x7f05a8104c30 2024-03-16T09:41:19.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.508+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a819fed0 con 0x7f05a8104c30 2024-03-16T09:41:19.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.509+0000 7f059f7fe640 1 -- 172.21.15.77:0/2581099356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0598005020 con 0x7f05a8104c30 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 Processor -- start 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 -- start start 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f059f7fe640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f05940a2890 0x7f05940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb7bd0 0x557bbedb7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedbd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedbd730 0x557bbedbfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557bbed1bc80 con 0x557bbedb7bd0 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557bbecc4aa0 con 0x557bbedb85c0 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05b2f22740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557bbed17d20 con 0x557bbedbd730 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05aec34640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f05940a2890 0x7f05940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f059f7fe640 1 -- 172.21.15.77:0/2581099356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f05980cefa0 con 0x7f05a8104c30 2024-03-16T09:41:19.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05ae433640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb7bd0 0x557bbedb7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05ae433640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb7bd0 0x557bbedb7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53020/0 (socket says 172.21.15.77:53020) 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.510+0000 7f05ae433640 1 -- 172.21.15.77:0/2527832839 learned_addr learned my addr 172.21.15.77:0/2527832839 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05adc32640 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedbd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05aec34640 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedbd730 0x557bbedbfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05ae433640 1 -- 172.21.15.77:0/2527832839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedbd730 msgr2=0x557bbedbfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05ae433640 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedbd730 0x557bbedbfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05ae433640 1 -- 172.21.15.77:0/2527832839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 msgr2=0x557bbedbd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05ae433640 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedbd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05ae433640 1 -- 172.21.15.77:0/2527832839 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbece6ff0 con 0x557bbedb7bd0 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05adc32640 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedbd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:19.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05aec34640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f05940a2890 0x7f05940a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f05a0005fd0 tx=0x7f05a0011bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f05ae433640 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb7bd0 0x557bbedb7ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009900 tx=0x7f05a4009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fff046dff343e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f059cff9640 1 -- 172.21.15.77:0/2527832839 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4045020 con 0x557bbedb7bd0 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f059cff9640 1 -- 172.21.15.77:0/2527832839 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4004040 con 0x557bbedb7bd0 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.511+0000 7f059cff9640 1 -- 172.21.15.77:0/2527832839 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4005540 con 0x557bbedb7bd0 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- 172.21.15.77:0/2527832839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb7bd0 msgr2=0x557bbedb7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb7bd0 0x557bbedb7ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009900 tx=0x7f05a4009560 comp rx=0 tx=0).stop 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- 172.21.15.77:0/2527832839 shutdown_connections 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedbd730 0x557bbedbfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedbd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 --2- 172.21.15.77:0/2527832839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb7bd0 0x557bbedb7ff0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- 172.21.15.77:0/2527832839 >> 172.21.15.77:0/2527832839 conn(0x557bbedb3140 msgr2=0x557bbedb3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- 172.21.15.77:0/2527832839 shutdown_connections 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- 172.21.15.77:0/2527832839 wait complete. 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- start start 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 Processor -- start 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedc2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557bbedc0680 con 0x557bbedbd730 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557bbecd7ef0 con 0x557bbedb85c0 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.512+0000 7f05b2f22740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557bbedc0800 con 0x557bbedb7bd0 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05b2f22740 10 client.0 _ll_register_callbacks cb 0x557bbecd2dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05aec34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedc2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05ae433640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedc2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:46422/0 (socket says 172.21.15.77:46422) 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05aec34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53034/0 (socket says 172.21.15.77:53034) 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 -- 172.21.15.77:0/1027663912 learned_addr learned my addr 172.21.15.77:0/1027663912 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 -- 172.21.15.77:0/1027663912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 msgr2=0x557bbedc1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 -- 172.21.15.77:0/1027663912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 msgr2=0x557bbedc9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 -- 172.21.15.77:0/1027663912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbece4a30 con 0x557bbedb85c0 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05aec34640 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05adc32640 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedc2210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05980981c0 tx=0x7f05980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5295551e49f1b41 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f0591ffb640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f0591ffb640 1 -- 172.21.15.77:0/1027663912 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980b3020 con 0x557bbedb85c0 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f0591ffb640 1 -- 172.21.15.77:0/1027663912 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f05980afbd0 con 0x557bbedb85c0 2024-03-16T09:41:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a4004040 con 0x557bbedb85c0 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f0591ffb640 1 -- 172.21.15.77:0/1027663912 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0598005680 con 0x557bbedb85c0 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05b2f22740 10 client.24965 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557bbea302d0 con 0x557bbedb85c0 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.513+0000 7f0591ffb640 1 -- 172.21.15.77:0/1027663912 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4004040 con 0x557bbedb85c0 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f0591ffb640 1 -- 172.21.15.77:0/1027663912 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f05980bb040 con 0x557bbedb85c0 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05b2f22740 10 client.24965 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05b2f22740 10 client.24965 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05b2f22740 20 client.24965 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbedce6d0 0x557bbedcfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05b2f22740 4 client.24965 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bbed291c0 con 0x557bbedce6d0 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05ae433640 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbedce6d0 0x557bbedcfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f0591ffb640 10 client.24965 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.514+0000 7f05ae433640 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbedce6d0 0x557bbedcfb40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b648a59eddc30521 server_cookie=16ea58a20b18ddfc in_seq=0 out_seq=0 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f0591ffb640 1 -- 172.21.15.77:0/1027663912 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557bbed291c0 con 0x557bbedce6d0 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f0591ffb640 10 client.24965 handle_command_reply: tid=0 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 client.24965 shutdown 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 msgr2=0x557bbedc2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedc2210 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05980981c0 tx=0x7f05980046c0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 shutdown_connections 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbedce6d0 0x557bbedcfb40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb85c0 0x557bbedc2210 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1027663912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f0591ffb640 0 client.24965 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 >> 172.21.15.77:0/1027663912 conn(0x557bbedb3140 msgr2=0x557bbedb2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 shutdown_connections 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 1 -- 172.21.15.77:0/1027663912 wait complete. 2024-03-16T09:41:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.515+0000 7f05b2f22740 20 client.24965 trim_cache size 0 max 16384 2024-03-16T09:41:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 Processor -- start 2024-03-16T09:41:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 -- start start 2024-03-16T09:41:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb7bd0 0x557bbedb3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb85c0 0x557bbedc9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557bbedcde80 con 0x557bbedbd730 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557bbedce550 con 0x557bbedb7bd0 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05b2f22740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557bbedcec20 con 0x557bbedb85c0 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53036/0 (socket says 172.21.15.77:53036) 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 -- 172.21.15.77:0/146355966 learned_addr learned my addr 172.21.15.77:0/146355966 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05adc32640 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb85c0 0x557bbedc9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05ae433640 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb7bd0 0x557bbedb3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 -- 172.21.15.77:0/146355966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb85c0 msgr2=0x557bbedc9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb85c0 0x557bbedc9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 -- 172.21.15.77:0/146355966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb7bd0 msgr2=0x557bbedb3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb7bd0 0x557bbedb3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 -- 172.21.15.77:0/146355966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbeafa6b0 con 0x557bbedbd730 2024-03-16T09:41:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.533+0000 7f05aec34640 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc0fc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f05a000fcb0 tx=0x7f05a0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbb2f9535770cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f0593fff640 1 -- 172.21.15.77:0/146355966 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a001a040 con 0x557bbedbd730 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f0593fff640 1 -- 172.21.15.77:0/146355966 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a0018040 con 0x557bbedbd730 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f0593fff640 1 -- 172.21.15.77:0/146355966 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0021430 con 0x557bbedbd730 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 -- 172.21.15.77:0/146355966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 msgr2=0x557bbedc0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc0fc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f05a000fcb0 tx=0x7f05a0009500 comp rx=0 tx=0).stop 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 -- 172.21.15.77:0/146355966 shutdown_connections 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedbd730 0x557bbedc0fc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb85c0 0x557bbedc9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 --2- 172.21.15.77:0/146355966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedb7bd0 0x557bbedb3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 -- 172.21.15.77:0/146355966 >> 172.21.15.77:0/146355966 conn(0x557bbedb3140 msgr2=0x557bbedc1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 -- 172.21.15.77:0/146355966 shutdown_connections 2024-03-16T09:41:19.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 -- 172.21.15.77:0/146355966 wait complete. 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 -- start start 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.534+0000 7f05b2f22740 1 Processor -- start 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb85c0 0x557bbedc48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedbd730 0x557bbedc78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557bbedcf9c0 con 0x557bbedb85c0 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557bbedcf2f0 con 0x557bbedbd730 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557bbedd0090 con 0x557bbedb7bd0 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 10 client.0 _ll_register_callbacks cb 0x7f0598004a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb85c0 0x557bbedc48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb85c0 0x557bbedc48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53046/0 (socket says 172.21.15.77:53046) 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 -- 172.21.15.77:0/1329453212 learned_addr learned my addr 172.21.15.77:0/1329453212 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05ae433640 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05aec34640 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedbd730 0x557bbedc78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 -- 172.21.15.77:0/1329453212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 msgr2=0x557bbedc4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 -- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedbd730 msgr2=0x557bbedc78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedbd730 0x557bbedc78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 -- 172.21.15.77:0/1329453212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbedc7ee0 con 0x557bbedb85c0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05adc32640 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb85c0 0x557bbedc48d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f05980a3a60 tx=0x7f05980048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa2a57c1fb35010 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f0591ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f0591ffb640 1 -- 172.21.15.77:0/1329453212 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980b8020 con 0x557bbedb85c0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f0591ffb640 1 -- 172.21.15.77:0/1329453212 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f059809ae50 con 0x557bbedb85c0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a0018040 con 0x557bbedb85c0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f0591ffb640 1 -- 172.21.15.77:0/1329453212 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f05980b3040 con 0x557bbedb85c0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 10 client.15081 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.535+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557bbedc91e0 con 0x557bbedb85c0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f0591ffb640 1 -- 172.21.15.77:0/1329453212 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a0018040 con 0x557bbedb85c0 2024-03-16T09:41:19.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f0591ffb640 1 -- 172.21.15.77:0/1329453212 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f05980bb070 con 0x557bbedb85c0 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05b2f22740 10 client.15081 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05b2f22740 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05b2f22740 20 client.15081 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbee38a10 0x557bbee37720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05b2f22740 4 client.15081 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "status"}) -- 0x557bbed23510 con 0x557bbee38a10 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05ae433640 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbee38a10 0x557bbee37720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f0591ffb640 10 client.15081 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05ae433640 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbee38a10 0x557bbee37720 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42f86729789959ca server_cookie=fdac742d1350b3ff in_seq=0 out_seq=0 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f0591ffb640 1 -- 172.21.15.77:0/1329453212 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557bbed23510 con 0x557bbee38a10 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f0591ffb640 10 client.15081 handle_command_reply: tid=0 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.536+0000 7f05b2f22740 1 client.15081 shutdown 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb85c0 msgr2=0x557bbedc48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb85c0 0x557bbedc48d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f05980a3a60 tx=0x7f05980048b0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 shutdown_connections 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x557bbee38a10 0x557bbee37720 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557bbedbd730 0x557bbedc78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f0591ffb640 0 client.15081 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557bbedb85c0 0x557bbedc48d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 --2- 172.21.15.77:0/1329453212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557bbedb7bd0 0x557bbedc4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 >> 172.21.15.77:0/1329453212 conn(0x557bbedb3140 msgr2=0x557bbedc0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 shutdown_connections 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 1 -- 172.21.15.77:0/1329453212 wait complete. 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b2f22740 20 client.15081 trim_cache size 0 max 16384 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:19.547 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24545, 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 83.358838578000004, 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 20, 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 88, 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 86, 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 88.933052357999998 2024-03-16T09:41:19.548 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f05940a2890 msgr2=0x7f05940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f05940a2890 0x7f05940a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f05a0005fd0 tx=0x7f05a0011bc0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8104c30 msgr2=0x7f05a8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8104c30 0x7f05a8198000 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f059800e9f0 tx=0x7f059800eeb0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 shutdown_connections 2024-03-16T09:41:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f05940a2890 0x7f05940a4d50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05a8107560 0x7f05a819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05a8104c30 0x7f05a8198000 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 --2- 172.21.15.77:0/2581099356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05a8102300 0x7f05a8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:19.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 >> 172.21.15.77:0/2581099356 conn(0x7f05a80fbc70 msgr2=0x7f05a8104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:19.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 shutdown_connections 2024-03-16T09:41:19.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:19.537+0000 7f05b0697640 1 -- 172.21.15.77:0/2581099356 wait complete. 2024-03-16T09:41:19.551 INFO:tasks.cephadm:Setup complete, yielding 2024-03-16T09:41:19.551 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T09:41:19.564 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:41:19.564 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-16T09:41:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:20.180 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:19 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1664711165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:20.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:19 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1664711165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:20.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:19 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1664711165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:21.166 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:20 smithi077 ceph-mon[32171]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:21.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:20 smithi119 ceph-mon[35632]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:21.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:20 smithi129 ceph-mon[35841]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 -- 172.21.15.77:0/2621784473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 msgr2=0x7f774c102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 --2- 172.21.15.77:0/2621784473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c102c20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7748009990 tx=0x7f7748009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 -- 172.21.15.77:0/2621784473 shutdown_connections 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 --2- 172.21.15.77:0/2621784473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c101e50 0x7f774c109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 --2- 172.21.15.77:0/2621784473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c102c20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 --2- 172.21.15.77:0/2621784473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f774c1014b0 0x7f774c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 -- 172.21.15.77:0/2621784473 >> 172.21.15.77:0/2621784473 conn(0x7f774c0fbf20 msgr2=0x7f774c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 -- 172.21.15.77:0/2621784473 shutdown_connections 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 -- 172.21.15.77:0/2621784473 wait complete. 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.541+0000 7f7753db7700 1 Processor -- start 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 -- start start 2024-03-16T09:41:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f774c1014b0 0x7f774c1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c101e50 0x7f774c197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f774c10ccc0 con 0x7f774c102800 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f774c10cb40 con 0x7f774c101e50 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f774c10ce40 con 0x7f774c1014b0 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c19e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53060/0 (socket says 172.21.15.77:53060) 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 -- 172.21.15.77:0/1672506236 learned_addr learned my addr 172.21.15.77:0/1672506236 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7751b53700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f774c1014b0 0x7f774c1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7751352700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c101e50 0x7f774c197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 -- 172.21.15.77:0/1672506236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f774c1014b0 msgr2=0x7f774c1973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f774c1014b0 0x7f774c1973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 -- 172.21.15.77:0/1672506236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c101e50 msgr2=0x7f774c197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c101e50 0x7f774c197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 -- 172.21.15.77:0/1672506236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774c1004a0 con 0x7f774c102800 2024-03-16T09:41:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7752354700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c19e980 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f774400ef30 tx=0x7f774400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7742ffd700 1 -- 172.21.15.77:0/1672506236 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77440092e0 con 0x7f774c102800 2024-03-16T09:41:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774c100730 con 0x7f774c102800 2024-03-16T09:41:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.542+0000 7f7742ffd700 1 -- 172.21.15.77:0/1672506236 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7744004bb0 con 0x7f774c102800 2024-03-16T09:41:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.543+0000 7f7742ffd700 1 -- 172.21.15.77:0/1672506236 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744004020 con 0x7f774c102800 2024-03-16T09:41:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.543+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774c100c20 con 0x7f774c102800 2024-03-16T09:41:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.543+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f774c068f90 con 0x7f774c102800 2024-03-16T09:41:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.545+0000 7f7742ffd700 1 -- 172.21.15.77:0/1672506236 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f774401b030 con 0x7f774c102800 2024-03-16T09:41:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.546+0000 7f7742ffd700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f77380a2920 0x7f77380a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.546+0000 7f7751b53700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f77380a2920 0x7f77380a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.546+0000 7f7742ffd700 1 -- 172.21.15.77:0/1672506236 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f77440cefc0 con 0x7f774c102800 2024-03-16T09:41:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.546+0000 7f7751b53700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f77380a2920 0x7f77380a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f774800b710 tx=0x7f7748005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:21.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.550+0000 7f7742ffd700 1 -- 172.21.15.77:0/1672506236 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7744022020 con 0x7f774c102800 2024-03-16T09:41:21.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.761+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f774c062710 con 0x7f77380a2920 2024-03-16T09:41:21.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.763+0000 7f7742ffd700 1 -- 172.21.15.77:0/1672506236 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f774c062710 con 0x7f77380a2920 2024-03-16T09:41:21.765 INFO:teuthology.orchestra.run.smithi077.stdout:Backend: cephadm 2024-03-16T09:41:21.766 INFO:teuthology.orchestra.run.smithi077.stdout:Available: Yes 2024-03-16T09:41:21.766 INFO:teuthology.orchestra.run.smithi077.stdout:Paused: No 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f77380a2920 msgr2=0x7f77380a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f77380a2920 0x7f77380a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f774800b710 tx=0x7f7748005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 msgr2=0x7f774c19e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c19e980 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f774400ef30 tx=0x7f774400c3a0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 shutdown_connections 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f77380a2920 0x7f77380a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f774c102800 0x7f774c19e980 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f774c1014b0 0x7f774c1973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 --2- 172.21.15.77:0/1672506236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c101e50 0x7f774c197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 >> 172.21.15.77:0/1672506236 conn(0x7f774c0fbf20 msgr2=0x7f774c105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 shutdown_connections 2024-03-16T09:41:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:21.765+0000 7f7753db7700 1 -- 172.21.15.77:0/1672506236 wait complete. 2024-03-16T09:41:22.425 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-16T09:41:22.665 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:23.093 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:22 smithi077 ceph-mon[32171]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:23.093 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:22 smithi077 ceph-mon[32171]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:41:23.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:22 smithi119 ceph-mon[35632]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:23.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:22 smithi119 ceph-mon[35632]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:41:23.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:22 smithi129 ceph-mon[35841]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:23.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:22 smithi129 ceph-mon[35841]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 -- 172.21.15.77:0/1544324124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 msgr2=0x7fcae8102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/1544324124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8102830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcae000b0d0 tx=0x7fcae000b3e0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 -- 172.21.15.77:0/1544324124 shutdown_connections 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/1544324124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcae81011d0 0x7fcae81015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/1544324124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102d70 0x7fcae8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/1544324124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8102830 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 -- 172.21.15.77:0/1544324124 >> 172.21.15.77:0/1544324124 conn(0x7fcae80fc770 msgr2=0x7fcae80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 -- 172.21.15.77:0/1544324124 shutdown_connections 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 -- 172.21.15.77:0/1544324124 wait complete. 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.299+0000 7fcaeef1b700 1 Processor -- start 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 -- start start 2024-03-16T09:41:24.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae81011d0 0x7fcae8197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcae8102d70 0x7fcae819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcae810cc40 con 0x7fcae81023d0 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcae810cac0 con 0x7fcae8102d70 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcae810cdc0 con 0x7fcae81011d0 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeccb7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae81011d0 0x7fcae8197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53084/0 (socket says 172.21.15.77:53084) 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 -- 172.21.15.77:0/3720578191 learned_addr learned my addr 172.21.15.77:0/3720578191 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaed4b8700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcae8102d70 0x7fcae819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 -- 172.21.15.77:0/3720578191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae81011d0 msgr2=0x7fcae8197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae81011d0 0x7fcae8197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 -- 172.21.15.77:0/3720578191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcae8102d70 msgr2=0x7fcae819e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcae8102d70 0x7fcae819e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 -- 172.21.15.77:0/3720578191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae819ef90 con 0x7fcae81023d0 2024-03-16T09:41:24.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae7fff700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8197870 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcae0007a50 tx=0x7fcae0007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:24.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae5ffb700 1 -- 172.21.15.77:0/3720578191 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0020070 con 0x7fcae81023d0 2024-03-16T09:41:24.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae8069f80 con 0x7fcae81023d0 2024-03-16T09:41:24.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.300+0000 7fcae5ffb700 1 -- 172.21.15.77:0/3720578191 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcae001ab70 con 0x7fcae81023d0 2024-03-16T09:41:24.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.301+0000 7fcae5ffb700 1 -- 172.21.15.77:0/3720578191 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0024a60 con 0x7fcae81023d0 2024-03-16T09:41:24.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.301+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae806a4b0 con 0x7fcae81023d0 2024-03-16T09:41:24.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.302+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae8068f90 con 0x7fcae81023d0 2024-03-16T09:41:24.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.303+0000 7fcae5ffb700 1 -- 172.21.15.77:0/3720578191 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fcae002b020 con 0x7fcae81023d0 2024-03-16T09:41:24.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.303+0000 7fcae5ffb700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcad00a2920 0x7fcad00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:24.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.303+0000 7fcaeccb7700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcad00a2920 0x7fcad00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:24.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.303+0000 7fcae5ffb700 1 -- 172.21.15.77:0/3720578191 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcae000f020 con 0x7fcae81023d0 2024-03-16T09:41:24.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.304+0000 7fcaeccb7700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcad00a2920 0x7fcad00a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcae8102230 tx=0x7fcad8007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:24.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.308+0000 7fcae5ffb700 1 -- 172.21.15.77:0/3720578191 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcae005f740 con 0x7fcae81023d0 2024-03-16T09:41:24.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.516+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fcae806a770 con 0x7fcad00a2920 2024-03-16T09:41:24.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.529+0000 7fcae5ffb700 1 -- 172.21.15.77:0/3720578191 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fcae806a770 con 0x7fcad00a2920 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.a smithi077 running (96s) 15s ago 96s 15.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 443f99b040ff 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.b smithi119 running (94s) 27s ago 93s 26.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ccf45c290e5a 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.c smithi129 running (107s) 27s ago 107s 18.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 6d306348933d 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.d smithi077 running (116s) 15s ago 115s 14.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d6b5984fc6ae 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.e smithi119 running (113s) 27s ago 112s 17.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 99588bee2562 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.f smithi129 running (110s) 27s ago 110s 18.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 8c6964a94547 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.g smithi077 running (99s) 15s ago 99s 13.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b8e1b98ba418 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.h smithi119 running (105s) 27s ago 104s 20.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 61a7a126692e 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.i smithi129 running (102s) 27s ago 102s 19.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 2569ce7aa2f2 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.j smithi077 running (118s) 15s ago 118s 13.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 20aa64164ac2 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.k smithi119 running (2m) 27s ago 2m 13.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 7002a90ce45b 2024-03-16T09:41:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:mds.l smithi129 running (2m) 27s ago 2m 13.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 292d32117075 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi077 *:9283,8765 running (9m) 15s ago 9m 512M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 4a5fa3c0118f 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi119 *:8443,8765 running (7m) 27s ago 7m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 60250e9ab4be 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.z smithi129 *:8443,8765 running (7m) 27s ago 7m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b bc3aef4b02e4 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (9m) 15s ago 9m 48.5M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 80a2c43e44ea 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi119 running (7m) 27s ago 7m 40.2M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 84d0a0b3dd42 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi129 running (7m) 27s ago 7m 39.7M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b a8d3da876c9c 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (6m) 15s ago 6m 48.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b e8d8bbc9ab77 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi119 running (6m) 27s ago 6m 48.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 3bd8f03faeaa 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi129 running (6m) 27s ago 6m 50.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8ffed0eb16ce 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (5m) 15s ago 5m 49.7M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 77ac324b72e3 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi119 running (5m) 27s ago 5m 52.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 5f1ef8507d74 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi129 running (4m) 27s ago 4m 55.1M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b f3fabb2baa13 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi077 running (4m) 15s ago 4m 47.3M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 95c568056b94 2024-03-16T09:41:24.534 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi119 running (4m) 27s ago 4m 47.2M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 54ef4f83b909 2024-03-16T09:41:24.535 INFO:teuthology.orchestra.run.smithi077.stdout:osd.8 smithi129 running (3m) 27s ago 3m 49.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b b53bbfde2470 2024-03-16T09:41:24.535 INFO:teuthology.orchestra.run.smithi077.stdout:osd.9 smithi077 running (3m) 15s ago 3m 46.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 4fca0808fd10 2024-03-16T09:41:24.535 INFO:teuthology.orchestra.run.smithi077.stdout:osd.10 smithi119 running (2m) 27s ago 2m 46.9M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 0da29803811c 2024-03-16T09:41:24.535 INFO:teuthology.orchestra.run.smithi077.stdout:osd.11 smithi129 running (2m) 27s ago 2m 50.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 25916333ee4e 2024-03-16T09:41:24.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcad00a2920 msgr2=0x7fcad00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcad00a2920 0x7fcad00a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcae8102230 tx=0x7fcad8007400 comp rx=0 tx=0).stop 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 msgr2=0x7fcae8197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8197870 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcae0007a50 tx=0x7fcae0007a80 comp rx=0 tx=0).stop 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 shutdown_connections 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcad00a2920 0x7fcad00a4de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcae8102d70 0x7fcae819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae81011d0 0x7fcae8197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 --2- 172.21.15.77:0/3720578191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcae81023d0 0x7fcae8197870 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 >> 172.21.15.77:0/3720578191 conn(0x7fcae80fc770 msgr2=0x7fcae81057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 shutdown_connections 2024-03-16T09:41:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:24.532+0000 7fcaeef1b700 1 -- 172.21.15.77:0/3720578191 wait complete. 2024-03-16T09:41:25.121 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:24 smithi077 ceph-mon[32171]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:25.121 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-16T09:41:25.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:24 smithi119 ceph-mon[35632]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:25.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:24 smithi129 ceph-mon[35841]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:25.359 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:26.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:25 smithi119 ceph-mon[35632]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:26.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:25 smithi077 ceph-mon[32171]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:26.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:25 smithi129 ceph-mon[35841]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 -- 172.21.15.77:0/2904219382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 msgr2=0x7f4e6c102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 --2- 172.21.15.77:0/2904219382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c102860 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4e600099c0 tx=0x7f4e60009cd0 comp rx=0 tx=0).stop 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 -- 172.21.15.77:0/2904219382 shutdown_connections 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 --2- 172.21.15.77:0/2904219382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e6c101200 0x7f4e6c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 --2- 172.21.15.77:0/2904219382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e6c102da0 0x7f4e6c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 --2- 172.21.15.77:0/2904219382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c102860 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 -- 172.21.15.77:0/2904219382 >> 172.21.15.77:0/2904219382 conn(0x7f4e6c0fc7a0 msgr2=0x7f4e6c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 -- 172.21.15.77:0/2904219382 shutdown_connections 2024-03-16T09:41:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 -- 172.21.15.77:0/2904219382 wait complete. 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 Processor -- start 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 -- start start 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e6c101200 0x7f4e6c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.896+0000 7f4e71518700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e71518700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e6c102da0 0x7f4e6c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e71518700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c10ccb0 con 0x7f4e6c102400 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e71518700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c10cb30 con 0x7f4e6c102da0 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e71518700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c10ce30 con 0x7f4e6c101200 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c1978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36570/0 (socket says 172.21.15.77:36570) 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 -- 172.21.15.77:0/3279401775 learned_addr learned my addr 172.21.15.77:0/3279401775 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6affd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e6c101200 0x7f4e6c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6b7fe700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e6c102da0 0x7f4e6c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 -- 172.21.15.77:0/3279401775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e6c101200 msgr2=0x7f4e6c197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e6c101200 0x7f4e6c197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 -- 172.21.15.77:0/3279401775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e6c102da0 msgr2=0x7f4e6c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e6c102da0 0x7f4e6c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 -- 172.21.15.77:0/3279401775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6c19efe0 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e6a7fc700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c1978c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000b580 tx=0x7f4e6000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e53fff700 1 -- 172.21.15.77:0/3279401775 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e60027070 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e6c19f270 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e53fff700 1 -- 172.21.15.77:0/3279401775 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e6000fd80 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e53fff700 1 -- 172.21.15.77:0/3279401775 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e600173f0 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.897+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e6c069f80 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.898+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e6c068f90 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.899+0000 7f4e53fff700 1 -- 172.21.15.77:0/3279401775 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4e6002a070 con 0x7f4e6c102400 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.900+0000 7f4e53fff700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4e540a4b30 0x7f4e540a6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.900+0000 7f4e6affd700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4e540a4b30 0x7f4e540a6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.900+0000 7f4e53fff700 1 -- 172.21.15.77:0/3279401775 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f4e600cf780 con 0x7f4e6c102400 2024-03-16T09:41:26.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.900+0000 7f4e6affd700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4e540a4b30 0x7f4e540a6ff0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c102260 tx=0x7f4e5c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:26.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:26.904+0000 7f4e53fff700 1 -- 172.21.15.77:0/3279401775 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e60013070 con 0x7f4e6c102400 2024-03-16T09:41:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:26 smithi077 ceph-mon[32171]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:27.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.110+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f4e6c1080d0 con 0x7f4e540a4b30 2024-03-16T09:41:27.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.116+0000 7f4e53fff700 1 -- 172.21.15.77:0/3279401775 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f4e6c1080d0 con 0x7f4e540a4b30 2024-03-16T09:41:27.118 INFO:teuthology.orchestra.run.smithi077.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-16T09:41:27.119 INFO:teuthology.orchestra.run.smithi077.stdout:mds.all 12/12 30s ago 2m smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi129=c;smithi129=f;smithi129=i;smithi129=l;count:12 2024-03-16T09:41:27.119 INFO:teuthology.orchestra.run.smithi077.stdout:mgr 3/3 30s ago 7m smithi077=x;smithi119=y;smithi129=z;count:3 2024-03-16T09:41:27.119 INFO:teuthology.orchestra.run.smithi077.stdout:mon 3/3 30s ago 7m smithi077:172.21.15.77=a;smithi119:172.21.15.119=b;smithi129:172.21.15.129=c;count:3 2024-03-16T09:41:27.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd 12 30s ago - 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4e540a4b30 msgr2=0x7f4e540a6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4e540a4b30 0x7f4e540a6ff0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c102260 tx=0x7f4e5c007400 comp rx=0 tx=0).stop 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 msgr2=0x7f4e6c1978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c1978c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000b580 tx=0x7f4e6000fa00 comp rx=0 tx=0).stop 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 shutdown_connections 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4e540a4b30 0x7f4e540a6ff0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e6c102da0 0x7f4e6c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e6c101200 0x7f4e6c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 --2- 172.21.15.77:0/3279401775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e6c102400 0x7f4e6c1978c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 >> 172.21.15.77:0/3279401775 conn(0x7f4e6c0fc7a0 msgr2=0x7f4e6c1057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 shutdown_connections 2024-03-16T09:41:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:27.118+0000 7f4e71518700 1 -- 172.21.15.77:0/3279401775 wait complete. 2024-03-16T09:41:27.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:26 smithi119 ceph-mon[35632]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:27.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:26 smithi129 ceph-mon[35841]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:27.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-16T09:41:27.993 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:29.107 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:28 smithi077 ceph-mon[32171]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:29.107 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:28 smithi077 ceph-mon[32171]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:29.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:28 smithi119 ceph-mon[35632]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:29.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:28 smithi119 ceph-mon[35632]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:29.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:28 smithi129 ceph-mon[35841]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:28 smithi129 ceph-mon[35841]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:29.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 -- 172.21.15.77:0/592188883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 msgr2=0x7f39e8107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 --2- 172.21.15.77:0/592188883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8107ae0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009990 tx=0x7f39dc009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 -- 172.21.15.77:0/592188883 shutdown_connections 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 --2- 172.21.15.77:0/592188883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39e8108020 0x7f39e810a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 --2- 172.21.15.77:0/592188883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39e8102dc0 0x7f39e81051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 --2- 172.21.15.77:0/592188883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8107ae0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 -- 172.21.15.77:0/592188883 >> 172.21.15.77:0/592188883 conn(0x7f39e80fc770 msgr2=0x7f39e80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 -- 172.21.15.77:0/592188883 shutdown_connections 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 -- 172.21.15.77:0/592188883 wait complete. 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.601+0000 7f39ed761700 1 Processor -- start 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39ed761700 1 -- start start 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39ed761700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39e8102dc0 0x7f39e81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39ed761700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39ed761700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39e8108020 0x7f39e819f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39ed761700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f39e810dd90 con 0x7f39e81056f0 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39ed761700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f39e810dc10 con 0x7f39e8108020 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39ed761700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39e810df10 con 0x7f39e8102dc0 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36582/0 (socket says 172.21.15.77:36582) 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 -- 172.21.15.77:0/1058146025 learned_addr learned my addr 172.21.15.77:0/1058146025 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e77fe700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39e8108020 0x7f39e819f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e6ffd700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39e8102dc0 0x7f39e81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 -- 172.21.15.77:0/1058146025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39e8102dc0 msgr2=0x7f39e81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39e8102dc0 0x7f39e81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 -- 172.21.15.77:0/1058146025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39e8108020 msgr2=0x7f39e819f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39e8108020 0x7f39e819f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 -- 172.21.15.77:0/1058146025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e81a0020 con 0x7f39e81056f0 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e77fe700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39e8108020 0x7f39e819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.602+0000 7f39e67fc700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8198900 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f39dc005910 tx=0x7f39dc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.603+0000 7f39cffff700 1 -- 172.21.15.77:0/1058146025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc026070 con 0x7f39e81056f0 2024-03-16T09:41:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.603+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e81a02b0 con 0x7f39e81056f0 2024-03-16T09:41:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.603+0000 7f39cffff700 1 -- 172.21.15.77:0/1058146025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39dc021410 con 0x7f39e81056f0 2024-03-16T09:41:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.603+0000 7f39cffff700 1 -- 172.21.15.77:0/1058146025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc017650 con 0x7f39e81056f0 2024-03-16T09:41:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.603+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e81a0760 con 0x7f39e81056f0 2024-03-16T09:41:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.604+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e8068f90 con 0x7f39e81056f0 2024-03-16T09:41:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.605+0000 7f39cffff700 1 -- 172.21.15.77:0/1058146025 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f39dc029070 con 0x7f39e81056f0 2024-03-16T09:41:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.605+0000 7f39cffff700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39d00a2890 0x7f39d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.605+0000 7f39e6ffd700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39d00a2890 0x7f39d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.605+0000 7f39cffff700 1 -- 172.21.15.77:0/1058146025 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f39dc0cffa0 con 0x7f39e81056f0 2024-03-16T09:41:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.606+0000 7f39e6ffd700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39d00a2890 0x7f39d00a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009c60 tx=0x7f39e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:29.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.609+0000 7f39cffff700 1 -- 172.21.15.77:0/1058146025 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f39dc030070 con 0x7f39e81056f0 2024-03-16T09:41:29.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.812+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f39e8062710 con 0x7f39d00a2890 2024-03-16T09:41:29.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.814+0000 7f39cffff700 1 -- 172.21.15.77:0/1058146025 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f39e8062710 con 0x7f39d00a2890 2024-03-16T09:41:29.817 INFO:teuthology.orchestra.run.smithi077.stdout:HOST ADDR LABELS STATUS 2024-03-16T09:41:29.817 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 172.21.15.77 2024-03-16T09:41:29.817 INFO:teuthology.orchestra.run.smithi077.stdout:smithi119 172.21.15.119 2024-03-16T09:41:29.817 INFO:teuthology.orchestra.run.smithi077.stdout:smithi129 172.21.15.129 2024-03-16T09:41:29.817 INFO:teuthology.orchestra.run.smithi077.stdout:3 hosts in cluster 2024-03-16T09:41:29.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39d00a2890 msgr2=0x7f39d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:29.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39d00a2890 0x7f39d00a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009c60 tx=0x7f39e0009260 comp rx=0 tx=0).stop 2024-03-16T09:41:29.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 msgr2=0x7f39e8198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:29.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8198900 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f39dc005910 tx=0x7f39dc009580 comp rx=0 tx=0).stop 2024-03-16T09:41:29.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 shutdown_connections 2024-03-16T09:41:29.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f39d00a2890 0x7f39d00a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39e8108020 0x7f39e819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39e8102dc0 0x7f39e81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 --2- 172.21.15.77:0/1058146025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e81056f0 0x7f39e8198900 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:29.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 >> 172.21.15.77:0/1058146025 conn(0x7f39e80fc770 msgr2=0x7f39e81038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:29.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 shutdown_connections 2024-03-16T09:41:29.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:29.816+0000 7f39ed761700 1 -- 172.21.15.77:0/1058146025 wait complete. 2024-03-16T09:41:30.139 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:29 smithi077 ceph-mon[32171]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T09:41:30.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:29 smithi119 ceph-mon[35632]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T09:41:30.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:29 smithi129 ceph-mon[35841]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T09:41:30.450 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-16T09:41:30.690 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:31.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:30 smithi077 ceph-mon[32171]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:31.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:30 smithi077 ceph-mon[32171]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:31.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:30 smithi119 ceph-mon[35632]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:31.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:30 smithi119 ceph-mon[35632]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:31.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:30 smithi129 ceph-mon[35841]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:30 smithi129 ceph-mon[35841]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1881117042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf8102d90 msgr2=0x7f7bf8109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1881117042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf8102d90 0x7f7bf8109640 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00099c0 tx=0x7f7bf0009cd0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1881117042 shutdown_connections 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1881117042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bf81011f0 0x7f7bf8101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1881117042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf8102d90 0x7f7bf8109640 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1881117042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bf81023f0 0x7f7bf8102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1881117042 >> 172.21.15.77:0/1881117042 conn(0x7f7bf80fc790 msgr2=0x7f7bf80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:32.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1881117042 shutdown_connections 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.299+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1881117042 wait complete. 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 Processor -- start 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 -- start start 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf81011f0 0x7f7bf8105180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bf81023f0 0x7f7bf81037d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bf8102d90 0x7f7bf8103d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf810cca0 con 0x7f7bf81011f0 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf810cb20 con 0x7f7bf81023f0 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bffe1a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf810ce20 con 0x7f7bf8102d90 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf81011f0 0x7f7bf8105180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf81011f0 0x7f7bf8105180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36598/0 (socket says 172.21.15.77:36598) 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 -- 172.21.15.77:0/1554312038 learned_addr learned my addr 172.21.15.77:0/1554312038 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfd3b5700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bf81023f0 0x7f7bf81037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:32.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfe3b7700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bf8102d90 0x7f7bf8103d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 -- 172.21.15.77:0/1554312038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bf8102d90 msgr2=0x7f7bf8103d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bf8102d90 0x7f7bf8103d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 -- 172.21.15.77:0/1554312038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bf81023f0 msgr2=0x7f7bf81037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bf81023f0 0x7f7bf81037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 -- 172.21.15.77:0/1554312038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf81045c0 con 0x7f7bf81011f0 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7bfdbb6700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf81011f0 0x7f7bf8105180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400efc0 tx=0x7f7bf400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.300+0000 7f7beeffd700 1 -- 172.21.15.77:0/1554312038 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf40092e0 con 0x7f7bf81011f0 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.301+0000 7f7beeffd700 1 -- 172.21.15.77:0/1554312038 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bf400c9c0 con 0x7f7bf81011f0 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.301+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf81a24d0 con 0x7f7bf81011f0 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.301+0000 7f7beeffd700 1 -- 172.21.15.77:0/1554312038 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4004570 con 0x7f7bf81011f0 2024-03-16T09:41:32.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.301+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf81a2a10 con 0x7f7bf81011f0 2024-03-16T09:41:32.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.301+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf8068f90 con 0x7f7bf81011f0 2024-03-16T09:41:32.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.302+0000 7f7beeffd700 1 -- 172.21.15.77:0/1554312038 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7bf401b030 con 0x7f7bf81011f0 2024-03-16T09:41:32.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.303+0000 7f7beeffd700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7be40a2910 0x7f7be40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:32.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.303+0000 7f7bfd3b5700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7be40a2910 0x7f7be40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:32.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.303+0000 7f7beeffd700 1 -- 172.21.15.77:0/1554312038 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7bf40ce860 con 0x7f7bf81011f0 2024-03-16T09:41:32.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.304+0000 7f7bfd3b5700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7be40a2910 0x7f7be40a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009c60 tx=0x7f7be8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:32.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.309+0000 7f7beeffd700 1 -- 172.21.15.77:0/1554312038 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7bf4022020 con 0x7f7bf81011f0 2024-03-16T09:41:32.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.512+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f7bf80fe930 con 0x7f7be40a2910 2024-03-16T09:41:32.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.513+0000 7f7beeffd700 1 -- 172.21.15.77:0/1554312038 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f7bf80fe930 con 0x7f7be40a2910 2024-03-16T09:41:32.516 INFO:teuthology.orchestra.run.smithi077.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-16T09:41:32.516 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T09:41:32.516 INFO:teuthology.orchestra.run.smithi077.stdout:smithi119 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y1400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-16T09:41:32.516 INFO:teuthology.orchestra.run.smithi077.stdout:smithi129 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300HY400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.515+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7be40a2910 msgr2=0x7f7be40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.515+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7be40a2910 0x7f7be40a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009c60 tx=0x7f7be8009260 comp rx=0 tx=0).stop 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.515+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf81011f0 msgr2=0x7f7bf8105180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.515+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf81011f0 0x7f7bf8105180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400efc0 tx=0x7f7bf400c3a0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 shutdown_connections 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7be40a2910 0x7f7be40a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bf8102d90 0x7f7bf8103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bf81011f0 0x7f7bf8105180 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 --2- 172.21.15.77:0/1554312038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bf81023f0 0x7f7bf81037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 >> 172.21.15.77:0/1554312038 conn(0x7f7bf80fc790 msgr2=0x7f7bf80fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 shutdown_connections 2024-03-16T09:41:32.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:32.516+0000 7f7bffe1a700 1 -- 172.21.15.77:0/1554312038 wait complete. 2024-03-16T09:41:33.104 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:32 smithi077 ceph-mon[32171]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:33.130 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T09:41:33.141 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:41:33.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-16T09:41:33.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:32 smithi119 ceph-mon[35632]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:33.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:32 smithi129 ceph-mon[35841]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:33.380 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:34.086 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:33 smithi077 ceph-mon[32171]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:34.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:33 smithi119 ceph-mon[35632]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:34.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:33 smithi129 ceph-mon[35841]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T09:41:35.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:34 smithi077 ceph-mon[32171]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:41:35.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:34 smithi119 ceph-mon[35632]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:41:35.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 -- 172.21.15.77:0/2910340327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc102410 msgr2=0x7f5ffc102870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:35.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 --2- 172.21.15.77:0/2910340327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc102410 0x7f5ffc102870 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0099c0 tx=0x7f5fec009cd0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 -- 172.21.15.77:0/2910340327 shutdown_connections 2024-03-16T09:41:35.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 --2- 172.21.15.77:0/2910340327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ffc101210 0x7f5ffc101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 --2- 172.21.15.77:0/2910340327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc102db0 0x7f5ffc109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 --2- 172.21.15.77:0/2910340327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc102410 0x7f5ffc102870 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 -- 172.21.15.77:0/2910340327 >> 172.21.15.77:0/2910340327 conn(0x7f5ffc0fc7d0 msgr2=0x7f5ffc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 -- 172.21.15.77:0/2910340327 shutdown_connections 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.211+0000 7f6004151700 1 -- 172.21.15.77:0/2910340327 wait complete. 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 Processor -- start 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 -- start start 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc101210 0x7f5ffc198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ffc102410 0x7f5ffc1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc102db0 0x7f5ffc19fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10cc80 con 0x7f5ffc101210 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10cb00 con 0x7f5ffc102410 2024-03-16T09:41:35.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6004151700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10ce00 con 0x7f5ffc102db0 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6001eed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc101210 0x7f5ffc198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6001eed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc101210 0x7f5ffc198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36626/0 (socket says 172.21.15.77:36626) 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f6001eed700 1 -- 172.21.15.77:0/1984232247 learned_addr learned my addr 172.21.15.77:0/1984232247 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f60016ec700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ffc102410 0x7f5ffc1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.212+0000 7f60026ee700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc102db0 0x7f5ffc19fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6001eed700 1 -- 172.21.15.77:0/1984232247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc102db0 msgr2=0x7f5ffc19fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6001eed700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc102db0 0x7f5ffc19fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6001eed700 1 -- 172.21.15.77:0/1984232247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ffc102410 msgr2=0x7f5ffc1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6001eed700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ffc102410 0x7f5ffc1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6001eed700 1 -- 172.21.15.77:0/1984232247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc1a00c0 con 0x7f5ffc101210 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f60016ec700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ffc102410 0x7f5ffc1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6001eed700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc101210 0x7f5ffc198460 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ff800e9a0 tx=0x7f5ff800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f5ff2ffd700 1 -- 172.21.15.77:0/1984232247 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff800cbd0 con 0x7f5ffc101210 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f5ff2ffd700 1 -- 172.21.15.77:0/1984232247 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ff80044d0 con 0x7f5ffc101210 2024-03-16T09:41:35.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ffc1a03b0 con 0x7f5ffc101210 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f5ff2ffd700 1 -- 172.21.15.77:0/1984232247 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8003e30 con 0x7f5ffc101210 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.213+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ffc1a08f0 con 0x7f5ffc101210 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.214+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc068f90 con 0x7f5ffc101210 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.215+0000 7f5ff2ffd700 1 -- 172.21.15.77:0/1984232247 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5ff8004000 con 0x7f5ffc101210 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.216+0000 7f5ff2ffd700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fe80a2890 0x7f5fe80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.216+0000 7f60016ec700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fe80a2890 0x7f5fe80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.216+0000 7f5ff2ffd700 1 -- 172.21.15.77:0/1984232247 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f5ff80cecb0 con 0x7f5ffc101210 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.216+0000 7f60016ec700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fe80a2890 0x7f5fe80a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000c00 tx=0x7f5fec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:35.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.220+0000 7f5ff2ffd700 1 -- 172.21.15.77:0/1984232247 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ff80042f0 con 0x7f5ffc101210 2024-03-16T09:41:35.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:34 smithi129 ceph-mon[35841]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:41:35.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.464+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f5ffc1a1320 con 0x7f5ffc101210 2024-03-16T09:41:35.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.465+0000 7f5ff2ffd700 1 -- 172.21.15.77:0/1984232247 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3054 (secure 0 0 0) 0x7f5ff809bd60 con 0x7f5ffc101210 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:e20 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:legacy client fscid: 1 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:Filesystem 'cephfs' (1) 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:fs_name cephfs 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 20 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-16T09:41:35.467 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-03-16T09:39:55.509822+0000 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:modified 2024-03-16T09:39:59.725431+0000 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:tableserver 0 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:root 0 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:session_timeout 300 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:session_autoclose 300 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:max_file_size 1099511627776 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:max_xattr_size 65536 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:required_client_features {} 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure 0 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure_osd_epoch 0 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:max_mds 3 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:in 0,1,2 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:up {0=24545,1=24503,2=24481} 2024-03-16T09:41:35.468 INFO:teuthology.orchestra.run.smithi077.stdout:failed 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:damaged 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:stopped 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:data_pools [3] 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:metadata_pool 2 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:inline_data disabled 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:balancer 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:bal_rank_mask -1 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:standby_count_wanted 0 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:qdb_cluster leader: 24545 members: 24481,24545,24503 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b{0:24545} state up:active seq 3 addr [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.e{0:24476} state up:standby-replay seq 1 addr [v2:172.21.15.119:6834/296042620,v1:172.21.15.119:6835/296042620] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.h{1:24503} state up:active seq 6 addr [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c{1:24469} state up:standby-replay seq 1 addr [v2:172.21.15.129:6836/814033341,v1:172.21.15.129:6837/814033341] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.i{2:24481} state up:active seq 6 addr [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.469 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.f{2:24457} state up:standby-replay seq 1 addr [v2:172.21.15.129:6834/2328435031,v1:172.21.15.129:6835/2328435031] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout:Standby daemons: 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.77:6834/1282436458,v1:172.21.15.77:6835/1282436458] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d{-1:14559} state up:standby seq 1 addr [v2:172.21.15.77:6836/3058701505,v1:172.21.15.77:6837/3058701505] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.g{-1:14613} state up:standby seq 1 addr [v2:172.21.15.77:6838/1432550583,v1:172.21.15.77:6839/1432550583] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a{-1:14625} state up:standby seq 1 addr [v2:172.21.15.77:6840/3585324919,v1:172.21.15.77:6841/3585324919] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.l{-1:24424} state up:standby seq 1 addr [v2:172.21.15.129:6832/3738546284,v1:172.21.15.129:6833/3738546284] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.470 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.k{-1:24437} state up:standby seq 1 addr [v2:172.21.15.119:6832/3136672591,v1:172.21.15.119:6833/3136672591] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T09:41:35.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fe80a2890 msgr2=0x7f5fe80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fe80a2890 0x7f5fe80a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000c00 tx=0x7f5fec005fb0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc101210 msgr2=0x7f5ffc198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc101210 0x7f5ffc198460 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ff800e9a0 tx=0x7f5ff800ecb0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 shutdown_connections 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5fe80a2890 0x7f5fe80a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc102db0 0x7f5ffc19fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ffc101210 0x7f5ffc198460 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 --2- 172.21.15.77:0/1984232247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ffc102410 0x7f5ffc1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 >> 172.21.15.77:0/1984232247 conn(0x7f5ffc0fc7d0 msgr2=0x7f5ffc0fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 shutdown_connections 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:35.467+0000 7f6004151700 1 -- 172.21.15.77:0/1984232247 wait complete. 2024-03-16T09:41:35.472 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-03-16T09:41:36.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:35 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1984232247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T09:41:36.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:35 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1984232247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T09:41:36.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-16T09:41:36.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:35 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1984232247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T09:41:36.445 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:37.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:36 smithi119 ceph-mon[35632]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:36 smithi077 ceph-mon[32171]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:37.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:36 smithi129 ceph-mon[35841]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:38.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 -- 172.21.15.77:0/829005436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc107ff0 msgr2=0x7f56dc10a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/829005436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc107ff0 0x7f56dc10a460 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009990 tx=0x7f56d4009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 -- 172.21.15.77:0/829005436 shutdown_connections 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/829005436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56dc102d90 0x7f56dc105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/829005436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc107ff0 0x7f56dc10a460 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/829005436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56dc1056c0 0x7f56dc107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 -- 172.21.15.77:0/829005436 >> 172.21.15.77:0/829005436 conn(0x7f56dc0fc7c0 msgr2=0x7f56dc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 -- 172.21.15.77:0/829005436 shutdown_connections 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 -- 172.21.15.77:0/829005436 wait complete. 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 Processor -- start 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.104+0000 7f56e3cdf700 1 -- start start 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56dc102d90 0x7f56dc198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:38.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc1056c0 0x7f56dc198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56dc107ff0 0x7f56dc19f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc10dd80 con 0x7f56dc1056c0 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc10dc00 con 0x7f56dc102d90 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc10df00 con 0x7f56dc107ff0 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc1056c0 0x7f56dc198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc1056c0 0x7f56dc198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59842/0 (socket says 172.21.15.77:59842) 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 -- 172.21.15.77:0/4178750420 learned_addr learned my addr 172.21.15.77:0/4178750420 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e1a7b700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56dc102d90 0x7f56dc198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e227c700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56dc107ff0 0x7f56dc19f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 -- 172.21.15.77:0/4178750420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56dc107ff0 msgr2=0x7f56dc19f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56dc107ff0 0x7f56dc19f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 -- 172.21.15.77:0/4178750420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56dc102d90 msgr2=0x7f56dc198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56dc102d90 0x7f56dc198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 -- 172.21.15.77:0/4178750420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56dc1a0080 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e227c700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56dc107ff0 0x7f56dc19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e127a700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc1056c0 0x7f56dc198960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f56d000e9f0 tx=0x7f56d000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56daffd700 1 -- 172.21.15.77:0/4178750420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d000cbd0 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56dc1a0370 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56daffd700 1 -- 172.21.15.77:0/4178750420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56d000cd70 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56daffd700 1 -- 172.21.15.77:0/4178750420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0005690 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.105+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56dc1a08b0 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.106+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56dc068f90 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.107+0000 7f56daffd700 1 -- 172.21.15.77:0/4178750420 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f56d0004750 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.108+0000 7f56daffd700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f56c40a2890 0x7f56c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.108+0000 7f56e1a7b700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f56c40a2890 0x7f56c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.108+0000 7f56daffd700 1 -- 172.21.15.77:0/4178750420 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f56d00cef50 con 0x7f56dc1056c0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.109+0000 7f56e1a7b700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f56c40a2890 0x7f56c40a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009da0 tx=0x7f56cc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:38.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.112+0000 7f56daffd700 1 -- 172.21.15.77:0/4178750420 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f56d0053740 con 0x7f56dc1056c0 2024-03-16T09:41:38.125 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:37 smithi077 ceph-mon[32171]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:41:38.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:37 smithi119 ceph-mon[35632]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:41:38.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:37 smithi129 ceph-mon[35841]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:41:38.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.306+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f56dc02db70 con 0x7f56dc1056c0 2024-03-16T09:41:38.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.307+0000 7f56daffd700 1 -- 172.21.15.77:0/4178750420 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5285 (secure 0 0 0) 0x7f56d009bfa0 con 0x7f56dc1056c0 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 88 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:fsid e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-03-16T09:32:02.879385+0000 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:modified 2024-03-16T09:40:58.857371+0000 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:crush_version 26 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:full_ratio 0.95 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:backfillfull_ratio 0.9 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:nearfull_ratio 0.85 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:require_min_compat_client luminous 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:min_compat_client luminous 2024-03-16T09:41:38.310 INFO:teuthology.orchestra.run.smithi077.stdout:require_osd_release squid 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:stretch_mode_enabled false 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:max_osd 12 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6802/3264923905,v1:172.21.15.77:6803/3264923905] [v2:172.21.15.77:6804/3264923905,v1:172.21.15.77:6805/3264923905] exists,up e6e86c30-0964-4336-a711-17623a2896cd 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6800/2451540128,v1:172.21.15.119:6801/2451540128] [v2:172.21.15.119:6802/2451540128,v1:172.21.15.119:6803/2451540128] exists,up 03eeab1c-5a6a-4a70-a48d-df6a2d9183cb 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6800/2888466595,v1:172.21.15.129:6801/2888466595] [v2:172.21.15.129:6802/2888466595,v1:172.21.15.129:6803/2888466595] exists,up b7e55cef-7750-4ba5-ab01-45d97769a4fb 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6810/698693547,v1:172.21.15.77:6811/698693547] [v2:172.21.15.77:6812/698693547,v1:172.21.15.77:6813/698693547] exists,up d3207723-0545-48aa-8c7d-468dbc1b316a 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 up in weight 1 up_from 33 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6808/3108480529,v1:172.21.15.119:6809/3108480529] [v2:172.21.15.119:6810/3108480529,v1:172.21.15.119:6811/3108480529] exists,up 43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 up in weight 1 up_from 39 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6808/2429107441,v1:172.21.15.129:6809/2429107441] [v2:172.21.15.129:6810/2429107441,v1:172.21.15.129:6811/2429107441] exists,up f0413402-bee8-4802-8b4a-d3a0a48049a1 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 up in weight 1 up_from 47 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6818/2647291293,v1:172.21.15.77:6819/2647291293] [v2:172.21.15.77:6820/2647291293,v1:172.21.15.77:6821/2647291293] exists,up a6700df2-9147-4ff5-93f7-105320d2a596 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 up in weight 1 up_from 52 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6816/2297351860,v1:172.21.15.119:6817/2297351860] [v2:172.21.15.119:6818/2297351860,v1:172.21.15.119:6819/2297351860] exists,up f717036c-9f74-49f7-9e6a-5a220593a044 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6816/3565481849,v1:172.21.15.129:6817/3565481849] [v2:172.21.15.129:6818/3565481849,v1:172.21.15.129:6819/3565481849] exists,up 02a6c20f-3922-494c-a45a-84173f7c41c4 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6826/1881713362,v1:172.21.15.77:6827/1881713362] [v2:172.21.15.77:6828/1881713362,v1:172.21.15.77:6829/1881713362] exists,up 8e592022-204a-4db0-b470-63978907639a 2024-03-16T09:41:38.311 INFO:teuthology.orchestra.run.smithi077.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6824/1354177519,v1:172.21.15.119:6825/1354177519] [v2:172.21.15.119:6826/1354177519,v1:172.21.15.119:6827/1354177519] exists,up 68f623f5-73ff-46d2-8ff8-30767522cc80 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6824/913848880,v1:172.21.15.129:6825/913848880] [v2:172.21.15.129:6826/913848880,v1:172.21.15.129:6827/913848880] exists,up ef748c28-d638-4d1d-a084-080ef7953c98 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 2.8 [2,11] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 2.e [2,4] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 2.f [8,11] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.2 [3,9] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.e [4,8] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.13 [4,9] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.24 [4,9] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.37 [5,6] 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/1569662437 expires 2024-03-17T09:32:57.693064+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/3524537702 expires 2024-03-17T09:32:57.693064+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6801/3741578273 expires 2024-03-17T09:32:57.693064+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/2291840305 expires 2024-03-17T09:32:29.772291+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/2778231529 expires 2024-03-17T09:32:29.772291+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6800/262452209 expires 2024-03-17T09:32:29.772291+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6801/262452209 expires 2024-03-17T09:32:29.772291+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/2049888475 expires 2024-03-17T09:32:57.693064+0000 2024-03-16T09:41:38.312 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/3919406383 expires 2024-03-17T09:32:29.772291+0000 2024-03-16T09:41:38.313 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6800/3741578273 expires 2024-03-17T09:32:57.693064+0000 2024-03-16T09:41:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f56c40a2890 msgr2=0x7f56c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f56c40a2890 0x7f56c40a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009da0 tx=0x7f56cc00b040 comp rx=0 tx=0).stop 2024-03-16T09:41:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc1056c0 msgr2=0x7f56dc198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc1056c0 0x7f56dc198960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f56d000e9f0 tx=0x7f56d000ed00 comp rx=0 tx=0).stop 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 shutdown_connections 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f56c40a2890 0x7f56c40a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56dc102d90 0x7f56dc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56dc107ff0 0x7f56dc19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 --2- 172.21.15.77:0/4178750420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56dc1056c0 0x7f56dc198960 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 >> 172.21.15.77:0/4178750420 conn(0x7f56dc0fc7c0 msgr2=0x7f56dc1036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 shutdown_connections 2024-03-16T09:41:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:38.309+0000 7f56e3cdf700 1 -- 172.21.15.77:0/4178750420 wait complete. 2024-03-16T09:41:38.900 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-16T09:41:38.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 -- 172.21.15.77:0/539512950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50100220 msgr2=0x7f8b50109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/539512950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50100220 0x7f8b50109dd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009ca0 tx=0x7f8b4802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 -- 172.21.15.77:0/539512950 shutdown_connections 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/539512950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50100220 0x7f8b50109dd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/539512950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b500ff860 0x7f8b500ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/539512950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b50102df0 0x7f8b50103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 -- 172.21.15.77:0/539512950 >> 172.21.15.77:0/539512950 conn(0x7f8b500fb400 msgr2=0x7f8b500fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 -- 172.21.15.77:0/539512950 shutdown_connections 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 -- 172.21.15.77:0/539512950 wait complete. 2024-03-16T09:41:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.012+0000 7f8b56c4b640 1 Processor -- start 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 -- start start 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b500ff860 0x7f8b501979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b50100220 0x7f8b50197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50102df0 0x7f8b5019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010d350 con 0x7f8b50102df0 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010d1d0 con 0x7f8b50100220 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010d4d0 con 0x7f8b500ff860 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50102df0 0x7f8b5019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50102df0 0x7f8b5019efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59846/0 (socket says 172.21.15.77:59846) 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 -- 172.21.15.77:0/3396043561 learned_addr learned my addr 172.21.15.77:0/3396043561 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b549e7640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b500ff860 0x7f8b501979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b4ffff640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b50100220 0x7f8b50197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 -- 172.21.15.77:0/3396043561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b500ff860 msgr2=0x7f8b501979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b500ff860 0x7f8b501979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 -- 172.21.15.77:0/3396043561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b50100220 msgr2=0x7f8b50197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b50100220 0x7f8b50197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 -- 172.21.15.77:0/3396043561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5019f610 con 0x7f8b50102df0 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b551e8640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50102df0 0x7f8b5019efa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b4802c700 tx=0x7f8b480328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b4dffb640 1 -- 172.21.15.77:0/3396043561 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48041020 con 0x7f8b50102df0 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b5019f8a0 con 0x7f8b50102df0 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b4dffb640 1 -- 172.21.15.77:0/3396043561 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b48003970 con 0x7f8b50102df0 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.013+0000 7f8b4dffb640 1 -- 172.21.15.77:0/3396043561 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48039040 con 0x7f8b50102df0 2024-03-16T09:41:39.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.014+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b5019fd90 con 0x7f8b50102df0 2024-03-16T09:41:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.015+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b50108d30 con 0x7f8b50102df0 2024-03-16T09:41:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.016+0000 7f8b4dffb640 1 -- 172.21.15.77:0/3396043561 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8b48044020 con 0x7f8b50102df0 2024-03-16T09:41:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.016+0000 7f8b4dffb640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b400a2800 0x7f8b400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.016+0000 7f8b549e7640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b400a2800 0x7f8b400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.016+0000 7f8b4dffb640 1 -- 172.21.15.77:0/3396043561 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8b480f0fb0 con 0x7f8b50102df0 2024-03-16T09:41:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.017+0000 7f8b549e7640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b400a2800 0x7f8b400a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b50069bb0 tx=0x7f8b44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.020+0000 7f8b4dffb640 1 -- 172.21.15.77:0/3396043561 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b4804b020 con 0x7f8b50102df0 2024-03-16T09:41:39.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:38 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4178750420' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T09:41:39.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.196+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b5002d380 con 0x7f8b50102df0 2024-03-16T09:41:39.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.197+0000 7f8b4dffb640 1 -- 172.21.15.77:0/3396043561 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f8b480be100 con 0x7f8b50102df0 2024-03-16T09:41:39.201 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:39.201 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:39.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.199+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b400a2800 msgr2=0x7f8b400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.199+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b400a2800 0x7f8b400a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b50069bb0 tx=0x7f8b44005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.199+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50102df0 msgr2=0x7f8b5019efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50102df0 0x7f8b5019efa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b4802c700 tx=0x7f8b480328a0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 shutdown_connections 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8b400a2800 0x7f8b400a4cc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50102df0 0x7f8b5019efa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b50100220 0x7f8b50197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 --2- 172.21.15.77:0/3396043561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b500ff860 0x7f8b501979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 >> 172.21.15.77:0/3396043561 conn(0x7f8b500fb400 msgr2=0x7f8b50105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 shutdown_connections 2024-03-16T09:41:39.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.200+0000 7f8b56c4b640 1 -- 172.21.15.77:0/3396043561 wait complete. 2024-03-16T09:41:39.212 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:38 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4178750420' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T09:41:39.214 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:41:39.214 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:41:39.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:38 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4178750420' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T09:41:39.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3972732664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41033a0 msgr2=0x7f1dd4109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3972732664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41033a0 0x7f1dd4109c30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009ca0 tx=0x7f1dcc02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3972732664 shutdown_connections 2024-03-16T09:41:39.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3972732664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41033a0 0x7f1dd4109c30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3972732664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd41029e0 0x7f1dd4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3972732664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd41017f0 0x7f1dd4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3972732664 >> 172.21.15.77:0/3972732664 conn(0x7f1dd40fcd80 msgr2=0x7f1dd40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3972732664 shutdown_connections 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.316+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3972732664 wait complete. 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 Processor -- start 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 -- start start 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd41017f0 0x7f1dd41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41029e0 0x7f1dd4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd41033a0 0x7f1dd419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd410d320 con 0x7f1dd41029e0 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd410d1a0 con 0x7f1dd41033a0 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd410d4a0 con 0x7f1dd41017f0 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41029e0 0x7f1dd4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41029e0 0x7f1dd4197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59856/0 (socket says 172.21.15.77:59856) 2024-03-16T09:41:39.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 -- 172.21.15.77:0/3289107660 learned_addr learned my addr 172.21.15.77:0/3289107660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dda885640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd41033a0 0x7f1dd419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 -- 172.21.15.77:0/3289107660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd41017f0 msgr2=0x7f1dd41979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd41017f0 0x7f1dd41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 -- 172.21.15.77:0/3289107660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd41033a0 msgr2=0x7f1dd419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd41033a0 0x7f1dd419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 -- 172.21.15.77:0/3289107660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd419f610 con 0x7f1dd41029e0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dda885640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd41033a0 0x7f1dd419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dd9883640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41029e0 0x7f1dd4197f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400e9f0 tx=0x7f1dc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dcb7fe640 1 -- 172.21.15.77:0/3289107660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc400cd80 con 0x7f1dd41029e0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dcb7fe640 1 -- 172.21.15.77:0/3289107660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dc40108e0 con 0x7f1dd41029e0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd419f900 con 0x7f1dd41029e0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1dcb7fe640 1 -- 172.21.15.77:0/3289107660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4018920 con 0x7f1dd41029e0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.317+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd419fe50 con 0x7f1dd41029e0 2024-03-16T09:41:39.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.318+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd4108c30 con 0x7f1dd41029e0 2024-03-16T09:41:39.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.319+0000 7f1dcb7fe640 1 -- 172.21.15.77:0/3289107660 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1dc4010a90 con 0x7f1dd41029e0 2024-03-16T09:41:39.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.320+0000 7f1dcb7fe640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1dc00a2800 0x7f1dc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.320+0000 7f1dda084640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1dc00a2800 0x7f1dc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.320+0000 7f1dcb7fe640 1 -- 172.21.15.77:0/3289107660 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1dc40ceea0 con 0x7f1dd41029e0 2024-03-16T09:41:39.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.320+0000 7f1dda084640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1dc00a2800 0x7f1dc00a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4102840 tx=0x7f1dd0009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.324+0000 7f1dcb7fe640 1 -- 172.21.15.77:0/3289107660 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1dc409bf50 con 0x7f1dd41029e0 2024-03-16T09:41:39.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.497+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1dd4062410 con 0x7f1dd41029e0 2024-03-16T09:41:39.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.498+0000 7f1dcb7fe640 1 -- 172.21.15.77:0/3289107660 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f1dc409b850 con 0x7f1dd41029e0 2024-03-16T09:41:39.501 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:41:39.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1dc00a2800 msgr2=0x7f1dc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1dc00a2800 0x7f1dc00a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4102840 tx=0x7f1dd0009630 comp rx=0 tx=0).stop 2024-03-16T09:41:39.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41029e0 msgr2=0x7f1dd4197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41029e0 0x7f1dd4197f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400e9f0 tx=0x7f1dc400eeb0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 shutdown_connections 2024-03-16T09:41:39.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1dc00a2800 0x7f1dc00a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd41033a0 0x7f1dd419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1dd41029e0 0x7f1dd4197f30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 --2- 172.21.15.77:0/3289107660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd41017f0 0x7f1dd41979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 >> 172.21.15.77:0/3289107660 conn(0x7f1dd40fcd80 msgr2=0x7f1dd4105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:39.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 shutdown_connections 2024-03-16T09:41:39.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.500+0000 7f1ddc2e8640 1 -- 172.21.15.77:0/3289107660 wait complete. 2024-03-16T09:41:39.514 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:41:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 -- 172.21.15.77:0/2583816588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181018e0 msgr2=0x7f2e18101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/2583816588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181018e0 0x7f2e18101d60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009900 tx=0x7f2e0c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:39.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 -- 172.21.15.77:0/2583816588 shutdown_connections 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/2583816588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e181022a0 0x7f2e18108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/2583816588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181018e0 0x7f2e18101d60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/2583816588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e181006f0 0x7f2e18100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 -- 172.21.15.77:0/2583816588 >> 172.21.15.77:0/2583816588 conn(0x7f2e180fbc80 msgr2=0x7f2e180fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 -- 172.21.15.77:0/2583816588 shutdown_connections 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.616+0000 7f2e1ea61640 1 -- 172.21.15.77:0/2583816588 wait complete. 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 Processor -- start 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 -- start start 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181006f0 0x7f2e18197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e181018e0 0x7f2e18197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e181022a0 0x7f2e1819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1810c200 con 0x7f2e181006f0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1810c080 con 0x7f2e181022a0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1ea61640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1810c380 con 0x7f2e181018e0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e17fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181006f0 0x7f2e18197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e17fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181006f0 0x7f2e18197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59880/0 (socket says 172.21.15.77:59880) 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e17fff640 1 -- 172.21.15.77:0/4203486175 learned_addr learned my addr 172.21.15.77:0/4203486175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e177fe640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e181018e0 0x7f2e18197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.617+0000 7f2e1cffe640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e181022a0 0x7f2e1819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e17fff640 1 -- 172.21.15.77:0/4203486175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e181018e0 msgr2=0x7f2e18197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e17fff640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e181018e0 0x7f2e18197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e17fff640 1 -- 172.21.15.77:0/4203486175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e181022a0 msgr2=0x7f2e1819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e17fff640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e181022a0 0x7f2e1819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e17fff640 1 -- 172.21.15.77:0/4203486175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1819f690 con 0x7f2e181006f0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e17fff640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181006f0 0x7f2e18197a70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800e9c0 tx=0x7f2e0800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e157fa640 1 -- 172.21.15.77:0/4203486175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0800ccc0 con 0x7f2e181006f0 2024-03-16T09:41:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1819f980 con 0x7f2e181006f0 2024-03-16T09:41:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e157fa640 1 -- 172.21.15.77:0/4203486175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e080035f0 con 0x7f2e181006f0 2024-03-16T09:41:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e157fa640 1 -- 172.21.15.77:0/4203486175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0800ce60 con 0x7f2e181006f0 2024-03-16T09:41:39.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.618+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1819fea0 con 0x7f2e181006f0 2024-03-16T09:41:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.619+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e181a00d0 con 0x7f2e181006f0 2024-03-16T09:41:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.620+0000 7f2e157fa640 1 -- 172.21.15.77:0/4203486175 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2e080037a0 con 0x7f2e181006f0 2024-03-16T09:41:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.621+0000 7f2e157fa640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2e000a2800 0x7f2e000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.621+0000 7f2e177fe640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2e000a2800 0x7f2e000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.621+0000 7f2e157fa640 1 -- 172.21.15.77:0/4203486175 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2e080cee00 con 0x7f2e181006f0 2024-03-16T09:41:39.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.621+0000 7f2e177fe640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2e000a2800 0x7f2e000a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009230 tx=0x7f2e0c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.625+0000 7f2e157fa640 1 -- 172.21.15.77:0/4203486175 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e0809beb0 con 0x7f2e181006f0 2024-03-16T09:41:39.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.800+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2e181a02a0 con 0x7f2e181006f0 2024-03-16T09:41:39.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.800+0000 7f2e157fa640 1 -- 172.21.15.77:0/4203486175 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f2e0809b7b0 con 0x7f2e181006f0 2024-03-16T09:41:39.803 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:41:39.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2e000a2800 msgr2=0x7f2e000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2e000a2800 0x7f2e000a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009230 tx=0x7f2e0c005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181006f0 msgr2=0x7f2e18197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181006f0 0x7f2e18197a70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800e9c0 tx=0x7f2e0800edc0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 shutdown_connections 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2e000a2800 0x7f2e000a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e181022a0 0x7f2e1819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e181018e0 0x7f2e18197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 --2- 172.21.15.77:0/4203486175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e181006f0 0x7f2e18197a70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 >> 172.21.15.77:0/4203486175 conn(0x7f2e180fbc80 msgr2=0x7f2e18069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 shutdown_connections 2024-03-16T09:41:39.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.802+0000 7f2e1ea61640 1 -- 172.21.15.77:0/4203486175 wait complete. 2024-03-16T09:41:39.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:41:39.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 -- 172.21.15.77:0/1145584039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901006e0 msgr2=0x7f4f90100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1145584039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901006e0 0x7f4f90100b00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009ca0 tx=0x7f4f8c02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 -- 172.21.15.77:0/1145584039 shutdown_connections 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1145584039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f90102290 0x7f4f90108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1145584039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f901018d0 0x7f4f90101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1145584039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901006e0 0x7f4f90100b00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 -- 172.21.15.77:0/1145584039 >> 172.21.15.77:0/1145584039 conn(0x7f4f900fbc70 msgr2=0x7f4f900fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.918+0000 7f4f97d62640 1 -- 172.21.15.77:0/1145584039 shutdown_connections 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 -- 172.21.15.77:0/1145584039 wait complete. 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 Processor -- start 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 -- start start 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f901006e0 0x7f4f90069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901018d0 0x7f4f9019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f90102290 0x7f4f901a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9010c1e0 con 0x7f4f901018d0 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9010c060 con 0x7f4f901006e0 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f97d62640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9010c360 con 0x7f4f90102290 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f952fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901018d0 0x7f4f9019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f952fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901018d0 0x7f4f9019eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59890/0 (socket says 172.21.15.77:59890) 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f952fd640 1 -- 172.21.15.77:0/1338454712 learned_addr learned my addr 172.21.15.77:0/1338454712 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f962ff640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f90102290 0x7f4f901a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.919+0000 7f4f95afe640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f901006e0 0x7f4f90069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f952fd640 1 -- 172.21.15.77:0/1338454712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f90102290 msgr2=0x7f4f901a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f952fd640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f90102290 0x7f4f901a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f952fd640 1 -- 172.21.15.77:0/1338454712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f901006e0 msgr2=0x7f4f90069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f952fd640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f901006e0 0x7f4f90069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f952fd640 1 -- 172.21.15.77:0/1338454712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f901a18c0 con 0x7f4f901018d0 2024-03-16T09:41:39.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f952fd640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901018d0 0x7f4f9019eb70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000d950 tx=0x7f4f8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f86ffd640 1 -- 172.21.15.77:0/1338454712 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80017030 con 0x7f4f901018d0 2024-03-16T09:41:39.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f86ffd640 1 -- 172.21.15.77:0/1338454712 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f80004020 con 0x7f4f901018d0 2024-03-16T09:41:39.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f901a1bb0 con 0x7f4f901018d0 2024-03-16T09:41:39.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f86ffd640 1 -- 172.21.15.77:0/1338454712 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f800047a0 con 0x7f4f901018d0 2024-03-16T09:41:39.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.920+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f901a2150 con 0x7f4f901018d0 2024-03-16T09:41:39.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.921+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f90107b70 con 0x7f4f901018d0 2024-03-16T09:41:39.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.922+0000 7f4f86ffd640 1 -- 172.21.15.77:0/1338454712 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4f800041d0 con 0x7f4f901018d0 2024-03-16T09:41:39.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.923+0000 7f4f86ffd640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4f7c0a2880 0x7f4f7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:39.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.923+0000 7f4f95afe640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4f7c0a2880 0x7f4f7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:39.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.923+0000 7f4f86ffd640 1 -- 172.21.15.77:0/1338454712 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f4f800cead0 con 0x7f4f901018d0 2024-03-16T09:41:39.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.923+0000 7f4f95afe640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4f7c0a2880 0x7f4f7c0a4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c032380 tx=0x7f4f8c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:39.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:39.927+0000 7f4f86ffd640 1 -- 172.21.15.77:0/1338454712 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f8009bc00 con 0x7f4f901018d0 2024-03-16T09:41:40.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.102+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4f9002d3b0 con 0x7f4f901018d0 2024-03-16T09:41:40.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.102+0000 7f4f86ffd640 1 -- 172.21.15.77:0/1338454712 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f4f8009b500 con 0x7f4f901018d0 2024-03-16T09:41:40.106 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:41:40.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4f7c0a2880 msgr2=0x7f4f7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4f7c0a2880 0x7f4f7c0a4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c032380 tx=0x7f4f8c000f40 comp rx=0 tx=0).stop 2024-03-16T09:41:40.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901018d0 msgr2=0x7f4f9019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901018d0 0x7f4f9019eb70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000d950 tx=0x7f4f8000de10 comp rx=0 tx=0).stop 2024-03-16T09:41:40.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 shutdown_connections 2024-03-16T09:41:40.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4f7c0a2880 0x7f4f7c0a4d40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f90102290 0x7f4f901a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f901018d0 0x7f4f9019eb70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 --2- 172.21.15.77:0/1338454712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f901006e0 0x7f4f90069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 >> 172.21.15.77:0/1338454712 conn(0x7f4f900fbc70 msgr2=0x7f4f90104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:40.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 shutdown_connections 2024-03-16T09:41:40.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.104+0000 7f4f97d62640 1 -- 172.21.15.77:0/1338454712 wait complete. 2024-03-16T09:41:40.117 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:39 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3396043561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:40.117 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:39 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3289107660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:40.118 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:39 smithi077 ceph-mon[32171]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:41:40.118 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:39 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4203486175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:40.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:41:40.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:39 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3396043561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:40.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:39 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3289107660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:40.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:39 smithi119 ceph-mon[35632]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:41:40.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:39 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4203486175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:40.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 -- 172.21.15.77:0/2687369215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 msgr2=0x7fad88102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 --2- 172.21.15.77:0/2687369215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88102e70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009900 tx=0x7fad7c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:40.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 -- 172.21.15.77:0/2687369215 shutdown_connections 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 --2- 172.21.15.77:0/2687369215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad881033b0 0x7fad88109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 --2- 172.21.15.77:0/2687369215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88102e70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 --2- 172.21.15.77:0/2687369215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad88101800 0x7fad88101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 -- 172.21.15.77:0/2687369215 >> 172.21.15.77:0/2687369215 conn(0x7fad880fcd90 msgr2=0x7fad880ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 -- 172.21.15.77:0/2687369215 shutdown_connections 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.220+0000 7fad8de63640 1 -- 172.21.15.77:0/2687369215 wait complete. 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 Processor -- start 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 -- start start 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad88101800 0x7fad88197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad881033b0 0x7fad8819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fad8810d310 con 0x7fad881029f0 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fad8810d190 con 0x7fad881033b0 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad8de63640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fad8810d490 con 0x7fad88101800 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad86ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59894/0 (socket says 172.21.15.77:59894) 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad86ffd640 1 -- 172.21.15.77:0/541618187 learned_addr learned my addr 172.21.15.77:0/541618187 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad87fff640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad881033b0 0x7fad8819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad877fe640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad88101800 0x7fad88197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad86ffd640 1 -- 172.21.15.77:0/541618187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad88101800 msgr2=0x7fad88197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad86ffd640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad88101800 0x7fad88197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad86ffd640 1 -- 172.21.15.77:0/541618187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad881033b0 msgr2=0x7fad8819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.221+0000 7fad86ffd640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad881033b0 0x7fad8819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.222+0000 7fad86ffd640 1 -- 172.21.15.77:0/541618187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8819f630 con 0x7fad881029f0 2024-03-16T09:41:40.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.222+0000 7fad86ffd640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88197f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fad7c006010 tx=0x7fad7c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:40.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.222+0000 7fad84ff9640 1 -- 172.21.15.77:0/541618187 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c043020 con 0x7fad881029f0 2024-03-16T09:41:40.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.222+0000 7fad84ff9640 1 -- 172.21.15.77:0/541618187 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad7c0043d0 con 0x7fad881029f0 2024-03-16T09:41:40.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.222+0000 7fad84ff9640 1 -- 172.21.15.77:0/541618187 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c003b80 con 0x7fad881029f0 2024-03-16T09:41:40.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.222+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8819f8c0 con 0x7fad881029f0 2024-03-16T09:41:40.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.222+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad8819fe10 con 0x7fad881029f0 2024-03-16T09:41:40.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.223+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad88108bf0 con 0x7fad881029f0 2024-03-16T09:41:40.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.224+0000 7fad84ff9640 1 -- 172.21.15.77:0/541618187 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fad7c046020 con 0x7fad881029f0 2024-03-16T09:41:40.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.224+0000 7fad84ff9640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad700a2800 0x7fad700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.224+0000 7fad877fe640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad700a2800 0x7fad700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.224+0000 7fad84ff9640 1 -- 172.21.15.77:0/541618187 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fad7c0f0c60 con 0x7fad881029f0 2024-03-16T09:41:40.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.225+0000 7fad877fe640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad700a2800 0x7fad700a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fad88102850 tx=0x7fad78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:40.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.229+0000 7fad84ff9640 1 -- 172.21.15.77:0/541618187 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad7c04d020 con 0x7fad881029f0 2024-03-16T09:41:40.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:39 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3396043561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:40.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:39 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3289107660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:40.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:39 smithi129 ceph-mon[35841]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:41:40.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:39 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4203486175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.439+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad880623e0 con 0x7fad881029f0 2024-03-16T09:41:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.439+0000 7fad84ff9640 1 -- 172.21.15.77:0/541618187 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11878 (secure 0 0 0) 0x7fad7c0bdd10 con 0x7fad881029f0 2024-03-16T09:41:40.442 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:39:59.725431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24545,24503]},"id":1}]} 2024-03-16T09:41:40.443 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-03-16T09:41:40.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad700a2800 msgr2=0x7fad700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad700a2800 0x7fad700a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fad88102850 tx=0x7fad78005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 msgr2=0x7fad88197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88197f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fad7c006010 tx=0x7fad7c005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 shutdown_connections 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fad700a2800 0x7fad700a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad881033b0 0x7fad8819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad881029f0 0x7fad88197f50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 --2- 172.21.15.77:0/541618187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad88101800 0x7fad88197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.441+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 >> 172.21.15.77:0/541618187 conn(0x7fad880fcd90 msgr2=0x7fad88105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.442+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 shutdown_connections 2024-03-16T09:41:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.442+0000 7fad8de63640 1 -- 172.21.15.77:0/541618187 wait complete. 2024-03-16T09:41:40.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:40.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- 172.21.15.77:0/4124937109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590100950 msgr2=0x7f7590100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 --2- 172.21.15.77:0/4124937109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590100950 0x7f7590100dd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f757c009900 tx=0x7f757c02bc80 comp rx=0 tx=0).stop 2024-03-16T09:41:40.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- 172.21.15.77:0/4124937109 shutdown_connections 2024-03-16T09:41:40.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 --2- 172.21.15.77:0/4124937109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590101310 0x7f7590108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 --2- 172.21.15.77:0/4124937109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590100950 0x7f7590100dd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 --2- 172.21.15.77:0/4124937109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7590101d90 0x7f75901021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- 172.21.15.77:0/4124937109 >> 172.21.15.77:0/4124937109 conn(0x7f75900fb400 msgr2=0x7f75900fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- 172.21.15.77:0/4124937109 shutdown_connections 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- 172.21.15.77:0/4124937109 wait complete. 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 Processor -- start 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- start start 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7590100950 0x7f75901003b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590101310 0x7f759019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590101d90 0x7f759019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f759010c260 con 0x7f7590101310 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f759010c0e0 con 0x7f7590100950 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.555+0000 7f7596ef1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f759010c3e0 con 0x7f7590101d90 2024-03-16T09:41:40.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590101310 0x7f759019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590101310 0x7f759019c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59902/0 (socket says 172.21.15.77:59902) 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 -- 172.21.15.77:0/2079913835 learned_addr learned my addr 172.21.15.77:0/2079913835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f7594c8d640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7590100950 0x7f75901003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f759548e640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590101d90 0x7f759019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 -- 172.21.15.77:0/2079913835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590101d90 msgr2=0x7f759019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590101d90 0x7f759019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 -- 172.21.15.77:0/2079913835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7590100950 msgr2=0x7f75901003b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7590100950 0x7f75901003b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 -- 172.21.15.77:0/2079913835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759019f720 con 0x7f7590101310 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758ffff640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590101310 0x7f759019c7f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f757c000c00 tx=0x7f757c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758dffb640 1 -- 172.21.15.77:0/2079913835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c043020 con 0x7f7590101310 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f759019f950 con 0x7f7590101310 2024-03-16T09:41:40.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758dffb640 1 -- 172.21.15.77:0/2079913835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f757c004350 con 0x7f7590101310 2024-03-16T09:41:40.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f758dffb640 1 -- 172.21.15.77:0/2079913835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c003b80 con 0x7f7590101310 2024-03-16T09:41:40.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.556+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f759019fea0 con 0x7f7590101310 2024-03-16T09:41:40.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.557+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7590068a20 con 0x7f7590101310 2024-03-16T09:41:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.558+0000 7f758dffb640 1 -- 172.21.15.77:0/2079913835 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f757c046020 con 0x7f7590101310 2024-03-16T09:41:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.559+0000 7f758dffb640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f75800a2800 0x7f75800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.559+0000 7f7594c8d640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f75800a2800 0x7f75800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.559+0000 7f758dffb640 1 -- 172.21.15.77:0/2079913835 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f757c0f0a80 con 0x7f7590101310 2024-03-16T09:41:40.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.559+0000 7f7594c8d640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f75800a2800 0x7f75800a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f75840095f0 tx=0x7f7584005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:40.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.564+0000 7f758dffb640 1 -- 172.21.15.77:0/2079913835 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f757c04d020 con 0x7f7590101310 2024-03-16T09:41:40.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.740+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75900623e0 con 0x7f7590101310 2024-03-16T09:41:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.742+0000 7f758dffb640 1 -- 172.21.15.77:0/2079913835 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f757c0bdb30 con 0x7f7590101310 2024-03-16T09:41:40.745 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:40.745 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f75800a2800 msgr2=0x7f75800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f75800a2800 0x7f75800a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f75840095f0 tx=0x7f7584005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590101310 msgr2=0x7f759019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590101310 0x7f759019c7f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f757c000c00 tx=0x7f757c005dc0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 shutdown_connections 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f75800a2800 0x7f75800a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590101d90 0x7f759019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7590101310 0x7f759019c7f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 --2- 172.21.15.77:0/2079913835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7590100950 0x7f75901003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 >> 172.21.15.77:0/2079913835 conn(0x7f75900fb400 msgr2=0x7f7590104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 shutdown_connections 2024-03-16T09:41:40.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.744+0000 7f7596ef1640 1 -- 172.21.15.77:0/2079913835 wait complete. 2024-03-16T09:41:40.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:41:40.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:41:40.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 -- 172.21.15.77:0/3831271643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 msgr2=0x7f86a40fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 --2- 172.21.15.77:0/3831271643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a40fff00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8694009930 tx=0x7f869402bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:40.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 -- 172.21.15.77:0/3831271643 shutdown_connections 2024-03-16T09:41:40.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 --2- 172.21.15.77:0/3831271643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a4103510 0x7f86a4109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 --2- 172.21.15.77:0/3831271643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86a4102b50 0x7f86a4102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 --2- 172.21.15.77:0/3831271643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a40fff00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 -- 172.21.15.77:0/3831271643 >> 172.21.15.77:0/3831271643 conn(0x7f86a40fb3d0 msgr2=0x7f86a40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:40.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 -- 172.21.15.77:0/3831271643 shutdown_connections 2024-03-16T09:41:40.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.860+0000 7f86a9be6640 1 -- 172.21.15.77:0/3831271643 wait complete. 2024-03-16T09:41:40.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 Processor -- start 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 -- start start 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a4102b50 0x7f86a4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86a4103510 0x7f86a419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86a410d340 con 0x7f86a40ffae0 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f86a410d1c0 con 0x7f86a4103510 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a9be6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f86a410d4c0 con 0x7f86a4102b50 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a4197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59924/0 (socket says 172.21.15.77:59924) 2024-03-16T09:41:40.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 -- 172.21.15.77:0/2535892019 learned_addr learned my addr 172.21.15.77:0/2535892019 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a2ffd640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a4102b50 0x7f86a4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a3fff640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86a4103510 0x7f86a419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 -- 172.21.15.77:0/2535892019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a4102b50 msgr2=0x7f86a4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a4102b50 0x7f86a4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 -- 172.21.15.77:0/2535892019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86a4103510 msgr2=0x7f86a419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86a4103510 0x7f86a419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.861+0000 7f86a37fe640 1 -- 172.21.15.77:0/2535892019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a419f6c0 con 0x7f86a40ffae0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.862+0000 7f86a37fe640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a4197aa0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8694000c00 tx=0x7f8694005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.862+0000 7f86a0ff9640 1 -- 172.21.15.77:0/2535892019 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694043020 con 0x7f86a40ffae0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.862+0000 7f86a0ff9640 1 -- 172.21.15.77:0/2535892019 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8694004350 con 0x7f86a40ffae0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.862+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a419f950 con 0x7f86a40ffae0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.862+0000 7f86a0ff9640 1 -- 172.21.15.77:0/2535892019 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694003b50 con 0x7f86a40ffae0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.862+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a419fea0 con 0x7f86a40ffae0 2024-03-16T09:41:40.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.863+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a4068a20 con 0x7f86a40ffae0 2024-03-16T09:41:40.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.864+0000 7f86a0ff9640 1 -- 172.21.15.77:0/2535892019 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8694046020 con 0x7f86a40ffae0 2024-03-16T09:41:40.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.864+0000 7f86a0ff9640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f868c0a2800 0x7f868c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:40.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.864+0000 7f86a2ffd640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f868c0a2800 0x7f868c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:40.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.864+0000 7f86a0ff9640 1 -- 172.21.15.77:0/2535892019 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f86940f0b60 con 0x7f86a40ffae0 2024-03-16T09:41:40.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.865+0000 7f86a2ffd640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f868c0a2800 0x7f868c0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f86a4198fb0 tx=0x7f8698005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:40.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:40.869+0000 7f86a0ff9640 1 -- 172.21.15.77:0/2535892019 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f869404d020 con 0x7f86a40ffae0 2024-03-16T09:41:41.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:40 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1338454712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:41.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:40 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/541618187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:41.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:40 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2079913835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:41.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.045+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f86a40623e0 con 0x7f86a40ffae0 2024-03-16T09:41:41.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.045+0000 7f86a0ff9640 1 -- 172.21.15.77:0/2535892019 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f86940bdc10 con 0x7f86a40ffae0 2024-03-16T09:41:41.048 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:41:41.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f868c0a2800 msgr2=0x7f868c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f868c0a2800 0x7f868c0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f86a4198fb0 tx=0x7f8698005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:41.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 msgr2=0x7f86a4197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a4197aa0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8694000c00 tx=0x7f8694005dc0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 shutdown_connections 2024-03-16T09:41:41.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f868c0a2800 0x7f868c0a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86a4103510 0x7f86a419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a4102b50 0x7f86a4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 --2- 172.21.15.77:0/2535892019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86a40ffae0 0x7f86a4197aa0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 >> 172.21.15.77:0/2535892019 conn(0x7f86a40fb3d0 msgr2=0x7f86a4105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:41.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 shutdown_connections 2024-03-16T09:41:41.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.047+0000 7f86a9be6640 1 -- 172.21.15.77:0/2535892019 wait complete. 2024-03-16T09:41:41.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:41:41.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 -- 172.21.15.77:0/2660593976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 msgr2=0x7ffa40101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 --2- 172.21.15.77:0/2660593976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40101d70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009900 tx=0x7ffa3002bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 -- 172.21.15.77:0/2660593976 shutdown_connections 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 --2- 172.21.15.77:0/2660593976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa401022b0 0x7ffa40108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 --2- 172.21.15.77:0/2660593976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40101d70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 --2- 172.21.15.77:0/2660593976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa40100700 0x7ffa40100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 -- 172.21.15.77:0/2660593976 >> 172.21.15.77:0/2660593976 conn(0x7ffa400fbc90 msgr2=0x7ffa400fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 -- 172.21.15.77:0/2660593976 shutdown_connections 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.162+0000 7ffa485b2640 1 -- 172.21.15.77:0/2660593976 wait complete. 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 Processor -- start 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 -- start start 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa40100700 0x7ffa40197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa401022b0 0x7ffa4019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010d350 con 0x7ffa401018f0 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010d1d0 con 0x7ffa40100700 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa485b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010d4d0 con 0x7ffa401022b0 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59932/0 (socket says 172.21.15.77:59932) 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 -- 172.21.15.77:0/4100816061 learned_addr learned my addr 172.21.15.77:0/4100816061 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa4634e640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa40100700 0x7ffa40197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa46b4f640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa401022b0 0x7ffa4019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 -- 172.21.15.77:0/4100816061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa401022b0 msgr2=0x7ffa4019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa401022b0 0x7ffa4019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 -- 172.21.15.77:0/4100816061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa40100700 msgr2=0x7ffa40197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa40100700 0x7ffa40197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 -- 172.21.15.77:0/4100816061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4019f670 con 0x7ffa401018f0 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa46b4f640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa401022b0 0x7ffa4019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa45b4d640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40197f90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ffa30006010 tx=0x7ffa30005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.163+0000 7ffa377fe640 1 -- 172.21.15.77:0/4100816061 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30043020 con 0x7ffa401018f0 2024-03-16T09:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.164+0000 7ffa377fe640 1 -- 172.21.15.77:0/4100816061 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa300043d0 con 0x7ffa401018f0 2024-03-16T09:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.164+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4019f900 con 0x7ffa401018f0 2024-03-16T09:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.164+0000 7ffa377fe640 1 -- 172.21.15.77:0/4100816061 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30003b80 con 0x7ffa401018f0 2024-03-16T09:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.164+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4019fe50 con 0x7ffa401018f0 2024-03-16T09:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.164+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa40107ae0 con 0x7ffa401018f0 2024-03-16T09:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.165+0000 7ffa377fe640 1 -- 172.21.15.77:0/4100816061 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ffa30046020 con 0x7ffa401018f0 2024-03-16T09:41:41.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.166+0000 7ffa377fe640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.166+0000 7ffa4634e640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.166+0000 7ffa377fe640 1 -- 172.21.15.77:0/4100816061 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ffa300f1300 con 0x7ffa401018f0 2024-03-16T09:41:41.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.166+0000 7ffa4634e640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa40101750 tx=0x7ffa3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:41.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.170+0000 7ffa377fe640 1 -- 172.21.15.77:0/4100816061 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa3004d020 con 0x7ffa401018f0 2024-03-16T09:41:41.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:40 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1338454712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:41.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:40 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/541618187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:40 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2079913835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:41.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:40 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1338454712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:41.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:40 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/541618187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:41.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:40 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2079913835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:41.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.342+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa4002d340 con 0x7ffa401018f0 2024-03-16T09:41:41.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.342+0000 7ffa377fe640 1 -- 172.21.15.77:0/4100816061 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7ffa300be3b0 con 0x7ffa401018f0 2024-03-16T09:41:41.345 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa2c0a2800 msgr2=0x7ffa2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa40101750 tx=0x7ffa3c005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 msgr2=0x7ffa40197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40197f90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ffa30006010 tx=0x7ffa30005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 shutdown_connections 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa401022b0 0x7ffa4019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa401018f0 0x7ffa40197f90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 --2- 172.21.15.77:0/4100816061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa40100700 0x7ffa40197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 >> 172.21.15.77:0/4100816061 conn(0x7ffa400fbc90 msgr2=0x7ffa40104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 shutdown_connections 2024-03-16T09:41:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.344+0000 7ffa485b2640 1 -- 172.21.15.77:0/4100816061 wait complete. 2024-03-16T09:41:41.358 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- 172.21.15.77:0/1063570206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41017f0 msgr2=0x7fd7a4101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1063570206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41017f0 0x7fd7a4101c10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009900 tx=0x7fd79c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- 172.21.15.77:0/1063570206 shutdown_connections 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1063570206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a41033a0 0x7fd7a4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1063570206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a41029e0 0x7fd7a4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1063570206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41017f0 0x7fd7a4101c10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- 172.21.15.77:0/1063570206 >> 172.21.15.77:0/1063570206 conn(0x7fd7a40fcd80 msgr2=0x7fd7a40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- 172.21.15.77:0/1063570206 shutdown_connections 2024-03-16T09:41:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- 172.21.15.77:0/1063570206 wait complete. 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 Processor -- start 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- start start 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a41017f0 0x7fd7a4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41029e0 0x7fd7a4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a41033a0 0x7fd7a419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a410d310 con 0x7fd7a41029e0 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a410d190 con 0x7fd7a41017f0 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.460+0000 7fd7abb10640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a410d490 con 0x7fd7a41033a0 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41029e0 0x7fd7a4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41029e0 0x7fd7a4197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59954/0 (socket says 172.21.15.77:59954) 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 -- 172.21.15.77:0/1391646569 learned_addr learned my addr 172.21.15.77:0/1391646569 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a98ac640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a41017f0 0x7fd7a4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7aa0ad640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a41033a0 0x7fd7a419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 -- 172.21.15.77:0/1391646569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a41033a0 msgr2=0x7fd7a419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a41033a0 0x7fd7a419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 -- 172.21.15.77:0/1391646569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a41017f0 msgr2=0x7fd7a4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a41017f0 0x7fd7a4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 -- 172.21.15.77:0/1391646569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a419f6b0 con 0x7fd7a41029e0 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7a90ab640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41029e0 0x7fd7a4197fd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd79400e9f0 tx=0x7fd79400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd79affd640 1 -- 172.21.15.77:0/1391646569 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79400cd80 con 0x7fd7a41029e0 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd79affd640 1 -- 172.21.15.77:0/1391646569 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7940054d0 con 0x7fd7a41029e0 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a419f9a0 con 0x7fd7a41029e0 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd79affd640 1 -- 172.21.15.77:0/1391646569 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794010490 con 0x7fd7a41029e0 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.461+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a419fef0 con 0x7fd7a41029e0 2024-03-16T09:41:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.462+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a4108ca0 con 0x7fd7a41029e0 2024-03-16T09:41:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.463+0000 7fd79affd640 1 -- 172.21.15.77:0/1391646569 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd794005020 con 0x7fd7a41029e0 2024-03-16T09:41:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.464+0000 7fd79affd640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd7900a2880 0x7fd7900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.464+0000 7fd7a98ac640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd7900a2880 0x7fd7900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.464+0000 7fd79affd640 1 -- 172.21.15.77:0/1391646569 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd7940cede0 con 0x7fd7a41029e0 2024-03-16T09:41:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.464+0000 7fd7a98ac640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd7900a2880 0x7fd7900a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005ab0 tx=0x7fd79c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.468+0000 7fd79affd640 1 -- 172.21.15.77:0/1391646569 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd79409be90 con 0x7fd7a41029e0 2024-03-16T09:41:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.644+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd7a40623e0 con 0x7fd7a41029e0 2024-03-16T09:41:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.645+0000 7fd79affd640 1 -- 172.21.15.77:0/1391646569 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fd79409b790 con 0x7fd7a41029e0 2024-03-16T09:41:41.647 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:41:41.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.646+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd7900a2880 msgr2=0x7fd7900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.646+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd7900a2880 0x7fd7900a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005ab0 tx=0x7fd79c005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.646+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41029e0 msgr2=0x7fd7a4197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.646+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41029e0 0x7fd7a4197fd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd79400e9f0 tx=0x7fd79400edf0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.646+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 shutdown_connections 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.647+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd7900a2880 0x7fd7900a4d40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.647+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7a41033a0 0x7fd7a419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.647+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a41029e0 0x7fd7a4197fd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.647+0000 7fd7abb10640 1 --2- 172.21.15.77:0/1391646569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a41017f0 0x7fd7a4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.647+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 >> 172.21.15.77:0/1391646569 conn(0x7fd7a40fcd80 msgr2=0x7fd7a4105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.647+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 shutdown_connections 2024-03-16T09:41:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.647+0000 7fd7abb10640 1 -- 172.21.15.77:0/1391646569 wait complete. 2024-03-16T09:41:41.660 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.760+0000 7f34cf93b640 1 -- 172.21.15.77:0/35170960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c81022d0 msgr2=0x7f34c8108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.760+0000 7f34cf93b640 1 --2- 172.21.15.77:0/35170960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c81022d0 0x7f34c8108b60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009ca0 tx=0x7f34c002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.760+0000 7f34cf93b640 1 -- 172.21.15.77:0/35170960 shutdown_connections 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.760+0000 7f34cf93b640 1 --2- 172.21.15.77:0/35170960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c81022d0 0x7f34c8108b60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.760+0000 7f34cf93b640 1 --2- 172.21.15.77:0/35170960 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c8101910 0x7f34c8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.760+0000 7f34cf93b640 1 --2- 172.21.15.77:0/35170960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34c8100720 0x7f34c8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.760+0000 7f34cf93b640 1 -- 172.21.15.77:0/35170960 >> 172.21.15.77:0/35170960 conn(0x7f34c80fbc90 msgr2=0x7f34c80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 -- 172.21.15.77:0/35170960 shutdown_connections 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 -- 172.21.15.77:0/35170960 wait complete. 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 Processor -- start 2024-03-16T09:41:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 -- start start 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c8100720 0x7f34c810b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34c8101910 0x7f34c81096e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c81022d0 0x7f34c8109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34c810d370 con 0x7f34c8100720 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f34c810d1f0 con 0x7f34c81022d0 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cf93b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34c810d4f0 con 0x7f34c8101910 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cd6d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c8100720 0x7f34c810b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cd6d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c8100720 0x7f34c810b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59960/0 (socket says 172.21.15.77:59960) 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cd6d7640 1 -- 172.21.15.77:0/3736276752 learned_addr learned my addr 172.21.15.77:0/3736276752 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cded8640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c81022d0 0x7f34c8109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.761+0000 7f34cced6640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34c8101910 0x7f34c81096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cd6d7640 1 -- 172.21.15.77:0/3736276752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34c8101910 msgr2=0x7f34c81096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cd6d7640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34c8101910 0x7f34c81096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cd6d7640 1 -- 172.21.15.77:0/3736276752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c81022d0 msgr2=0x7f34c8109c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cd6d7640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c81022d0 0x7f34c8109c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cd6d7640 1 -- 172.21.15.77:0/3736276752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c810a4b0 con 0x7f34c8100720 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cced6640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34c8101910 0x7f34c81096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cd6d7640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c8100720 0x7f34c810b0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f34b800e9c0 tx=0x7f34b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34be7fc640 1 -- 172.21.15.77:0/3736276752 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b800cd80 con 0x7f34c8100720 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34be7fc640 1 -- 172.21.15.77:0/3736276752 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34b80035f0 con 0x7f34c8100720 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c81a1b60 con 0x7f34c8100720 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34be7fc640 1 -- 172.21.15.77:0/3736276752 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b8010690 con 0x7f34c8100720 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.762+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c81a20b0 con 0x7f34c8100720 2024-03-16T09:41:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.763+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c8107b60 con 0x7f34c8100720 2024-03-16T09:41:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.764+0000 7f34be7fc640 1 -- 172.21.15.77:0/3736276752 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f34b80037a0 con 0x7f34c8100720 2024-03-16T09:41:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.764+0000 7f34be7fc640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34b40a2880 0x7f34b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.764+0000 7f34cced6640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34b40a2880 0x7f34b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.764+0000 7f34be7fc640 1 -- 172.21.15.77:0/3736276752 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f34b80cf020 con 0x7f34c8100720 2024-03-16T09:41:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.765+0000 7f34cced6640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34b40a2880 0x7f34b40a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f34c810ae10 tx=0x7f34c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.769+0000 7f34be7fc640 1 -- 172.21.15.77:0/3736276752 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f34b809c0d0 con 0x7f34c8100720 2024-03-16T09:41:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.986+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34c80623e0 con 0x7f34c8100720 2024-03-16T09:41:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.987+0000 7f34be7fc640 1 -- 172.21.15.77:0/3736276752 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11878 (secure 0 0 0) 0x7f34b809b9d0 con 0x7f34c8100720 2024-03-16T09:41:41.990 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:41.990 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:39:59.725431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24545,24503]},"id":1}]} 2024-03-16T09:41:41.990 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2024-03-16T09:41:41.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34b40a2880 msgr2=0x7f34b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34b40a2880 0x7f34b40a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f34c810ae10 tx=0x7f34c4005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c8100720 msgr2=0x7f34c810b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c8100720 0x7f34c810b0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f34b800e9c0 tx=0x7f34b800ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 shutdown_connections 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34b40a2880 0x7f34b40a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c81022d0 0x7f34c8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34c8101910 0x7f34c81096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 --2- 172.21.15.77:0/3736276752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34c8100720 0x7f34c810b0a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 >> 172.21.15.77:0/3736276752 conn(0x7f34c80fbc90 msgr2=0x7f34c8104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 shutdown_connections 2024-03-16T09:41:41.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:41.989+0000 7f34cf93b640 1 -- 172.21.15.77:0/3736276752 wait complete. 2024-03-16T09:41:42.003 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 -- 172.21.15.77:0/2110771169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394101970 msgr2=0x7f8394101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 --2- 172.21.15.77:0/2110771169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394101970 0x7f8394101df0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8388009900 tx=0x7f838802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 -- 172.21.15.77:0/2110771169 shutdown_connections 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 --2- 172.21.15.77:0/2110771169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8394102330 0x7f8394108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 --2- 172.21.15.77:0/2110771169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394101970 0x7f8394101df0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 --2- 172.21.15.77:0/2110771169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8394100780 0x7f8394100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 -- 172.21.15.77:0/2110771169 >> 172.21.15.77:0/2110771169 conn(0x7f83940fbd10 msgr2=0x7f83940fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 -- 172.21.15.77:0/2110771169 shutdown_connections 2024-03-16T09:41:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.102+0000 7f839884b640 1 -- 172.21.15.77:0/2110771169 wait complete. 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 Processor -- start 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 -- start start 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394100780 0x7f8394197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8394101970 0x7f8394197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8394102330 0x7f839419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f839410d360 con 0x7f8394100780 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f839410d1e0 con 0x7f8394101970 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f839410d4e0 con 0x7f8394102330 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394100780 0x7f8394197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394100780 0x7f8394197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59968/0 (socket says 172.21.15.77:59968) 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 -- 172.21.15.77:0/2025065647 learned_addr learned my addr 172.21.15.77:0/2025065647 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f8391d9b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8394101970 0x7f8394197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f8392d9d640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8394102330 0x7f839419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 -- 172.21.15.77:0/2025065647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8394102330 msgr2=0x7f839419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8394102330 0x7f839419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 -- 172.21.15.77:0/2025065647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8394101970 msgr2=0x7f8394197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8394101970 0x7f8394197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 -- 172.21.15.77:0/2025065647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839419f640 con 0x7f8394100780 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f8392d9d640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8394102330 0x7f839419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839259c640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394100780 0x7f8394197a20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f838400e9c0 tx=0x7f838400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f83837fe640 1 -- 172.21.15.77:0/2025065647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838400cd80 con 0x7f8394100780 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f83837fe640 1 -- 172.21.15.77:0/2025065647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83840035f0 con 0x7f8394100780 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f83837fe640 1 -- 172.21.15.77:0/2025065647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384010690 con 0x7f8394100780 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839419f930 con 0x7f8394100780 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.103+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839419fea0 con 0x7f8394100780 2024-03-16T09:41:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.104+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8394107b10 con 0x7f8394100780 2024-03-16T09:41:42.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.105+0000 7f83837fe640 1 -- 172.21.15.77:0/2025065647 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f83840037a0 con 0x7f8394100780 2024-03-16T09:41:42.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.106+0000 7f83837fe640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f837c0a2880 0x7f837c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.106+0000 7f8391d9b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f837c0a2880 0x7f837c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.106+0000 7f83837fe640 1 -- 172.21.15.77:0/2025065647 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f83840cef70 con 0x7f8394100780 2024-03-16T09:41:42.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.106+0000 7f8391d9b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f837c0a2880 0x7f837c0a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8388009230 tx=0x7f8388005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.109+0000 7f83837fe640 1 -- 172.21.15.77:0/2025065647 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f838409c020 con 0x7f8394100780 2024-03-16T09:41:42.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:41 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2535892019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:41 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4100816061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:41 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1391646569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:41 smithi077 ceph-mon[32171]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:42.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.287+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f839402d420 con 0x7f8394100780 2024-03-16T09:41:42.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.288+0000 7f83837fe640 1 -- 172.21.15.77:0/2025065647 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f838409b920 con 0x7f8394100780 2024-03-16T09:41:42.291 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:42.291 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:42.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f837c0a2880 msgr2=0x7f837c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f837c0a2880 0x7f837c0a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8388009230 tx=0x7f8388005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394100780 msgr2=0x7f8394197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394100780 0x7f8394197a20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f838400e9c0 tx=0x7f838400ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 shutdown_connections 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f837c0a2880 0x7f837c0a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8394102330 0x7f839419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8394101970 0x7f8394197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 --2- 172.21.15.77:0/2025065647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8394100780 0x7f8394197a20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 >> 172.21.15.77:0/2025065647 conn(0x7f83940fbd10 msgr2=0x7f8394104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 shutdown_connections 2024-03-16T09:41:42.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.290+0000 7f839884b640 1 -- 172.21.15.77:0/2025065647 wait complete. 2024-03-16T09:41:42.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:41 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2535892019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:41 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4100816061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:41 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1391646569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:41 smithi129 ceph-mon[35841]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:42.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T09:39:55.509822+0000', 'modified': '2024-03-16T09:39:59.725431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24545, 'mds_1': 24503, 'mds_2': 24481}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6835/2328435031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 2328435031}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 2328435031}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.129:6837/814033341', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 814033341}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 814033341}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/296042620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 296042620}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 296042620}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6839/1514269525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 1514269525}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 1514269525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.119:6837/1948234265', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 1948234265}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 1948234265}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/3029827381', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 3029827381}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 3029827381}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24545, 'qdb_cluster': [24481, 24545, 24503]} 2024-03-16T09:41:42.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T09:41:42.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T09:41:42.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.405+0000 7fb060475640 1 -- 172.21.15.77:0/700402680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581006e0 msgr2=0x7fb058100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 --2- 172.21.15.77:0/700402680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581006e0 0x7fb058100b00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb054009ca0 tx=0x7fb05402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- 172.21.15.77:0/700402680 shutdown_connections 2024-03-16T09:41:42.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 --2- 172.21.15.77:0/700402680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb058102290 0x7fb058108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 --2- 172.21.15.77:0/700402680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0581018d0 0x7fb058101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 --2- 172.21.15.77:0/700402680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581006e0 0x7fb058100b00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- 172.21.15.77:0/700402680 >> 172.21.15.77:0/700402680 conn(0x7fb0580fbc70 msgr2=0x7fb0580fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- 172.21.15.77:0/700402680 shutdown_connections 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- 172.21.15.77:0/700402680 wait complete. 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 Processor -- start 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- start start 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0581006e0 0x7fb0580694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581018d0 0x7fb05819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb058102290 0x7fb05819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb05810c200 con 0x7fb0581018d0 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb05810c080 con 0x7fb058102290 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb060475640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb05810c380 con 0x7fb0581006e0 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb05da10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581018d0 0x7fb05819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb05da10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581018d0 0x7fb05819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59980/0 (socket says 172.21.15.77:59980) 2024-03-16T09:41:42.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb05da10640 1 -- 172.21.15.77:0/1905283702 learned_addr learned my addr 172.21.15.77:0/1905283702 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb05e211640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0581006e0 0x7fb0580694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.406+0000 7fb05ea12640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb058102290 0x7fb05819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb05da10640 1 -- 172.21.15.77:0/1905283702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0581006e0 msgr2=0x7fb0580694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb05da10640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0581006e0 0x7fb0580694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb05da10640 1 -- 172.21.15.77:0/1905283702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb058102290 msgr2=0x7fb05819ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb05da10640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb058102290 0x7fb05819ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb05da10640 1 -- 172.21.15.77:0/1905283702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05819f710 con 0x7fb0581018d0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb05da10640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581018d0 0x7fb05819c810 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb04800e9f0 tx=0x7fb04800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb04f7fe640 1 -- 172.21.15.77:0/1905283702 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04800ccc0 con 0x7fb0581018d0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb04f7fe640 1 -- 172.21.15.77:0/1905283702 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb04800ce60 con 0x7fb0581018d0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb04f7fe640 1 -- 172.21.15.77:0/1905283702 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048010430 con 0x7fb0581018d0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05819fa00 con 0x7fb0581018d0 2024-03-16T09:41:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.407+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05819ff50 con 0x7fb0581018d0 2024-03-16T09:41:42.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.409+0000 7fb04f7fe640 1 -- 172.21.15.77:0/1905283702 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb048005020 con 0x7fb0581018d0 2024-03-16T09:41:42.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.409+0000 7fb06305c740 1 Processor -- start 2024-03-16T09:41:42.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb06305c740 1 -- start start 2024-03-16T09:41:42.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb06305c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badae5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb06305c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae59d0 0x559badaea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb06305c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badaeab40 0x559badaecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb06305c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559bada44c80 con 0x559badae59d0 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb06305c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559bad9edaa0 con 0x559badae4fe0 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb06305c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559bada40d20 con 0x559badaeab40 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb04f7fe640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0440a2880 0x7fb0440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae59d0 0x559badaea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae59d0 0x559badaea4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59988/0 (socket says 172.21.15.77:59988) 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 -- 172.21.15.77:0/471891349 learned_addr learned my addr 172.21.15.77:0/471891349 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05da10640 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badaeab40 0x559badaecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05ea12640 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badae5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05ea12640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0440a2880 0x7fb0440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb04f7fe640 1 -- 172.21.15.77:0/1905283702 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb0480ceda0 con 0x7fb0581018d0 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 -- 172.21.15.77:0/471891349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badaeab40 msgr2=0x559badaecfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badaeab40 0x559badaecfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 -- 172.21.15.77:0/471891349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 msgr2=0x559badae5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badae5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05e211640 1 -- 172.21.15.77:0/471891349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bada0fff0 con 0x559badae59d0 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.410+0000 7fb05ea12640 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badae5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.411+0000 7fb05e211640 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae59d0 0x559badaea4d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb054009ca0 tx=0x7fb054005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf8248e90733076 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.411+0000 7fb05ea12640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0440a2880 0x7fb0440a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x559badae6230 tx=0x7fb050005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.411+0000 7fb04cbfe640 1 -- 172.21.15.77:0/471891349 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054043070 con 0x559badae59d0 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.411+0000 7fb04cbfe640 1 -- 172.21.15.77:0/471891349 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb054032db0 con 0x559badae59d0 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.411+0000 7fb04cbfe640 1 -- 172.21.15.77:0/471891349 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054039040 con 0x559badae59d0 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.411+0000 7fb06305c740 1 -- 172.21.15.77:0/471891349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae59d0 msgr2=0x559badaea4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.411+0000 7fb06305c740 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae59d0 0x559badaea4d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb054009ca0 tx=0x7fb054005f70 comp rx=0 tx=0).stop 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- 172.21.15.77:0/471891349 shutdown_connections 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badaeab40 0x559badaecfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae59d0 0x559badaea4d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 --2- 172.21.15.77:0/471891349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badae5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- 172.21.15.77:0/471891349 >> 172.21.15.77:0/471891349 conn(0x559badadc140 msgr2=0x559badadc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- 172.21.15.77:0/471891349 shutdown_connections 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- 172.21.15.77:0/471891349 wait complete. 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- start start 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 Processor -- start 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badaef130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaef670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559badaeda90 con 0x559badaeab40 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559bada00ef0 con 0x559badae4fe0 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559badaedc10 con 0x559badae59d0 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb06305c740 10 client.0 _ll_register_callbacks cb 0x559bad9fbdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb05ea12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb05da10640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaef670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb05ea12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59996/0 (socket says 172.21.15.77:59996) 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb05ea12640 1 -- 172.21.15.77:0/2676066680 learned_addr learned my addr 172.21.15.77:0/2676066680 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.412+0000 7fb05e211640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badaef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05ea12640 1 -- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 msgr2=0x559badaef670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05ea12640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaef670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05ea12640 1 -- 172.21.15.77:0/2676066680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 msgr2=0x559badaef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05ea12640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badaef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05ea12640 1 -- 172.21.15.77:0/2676066680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bada0da30 con 0x559badaeab40 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05da10640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaef670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05e211640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badaef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb05ea12640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf66b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb050004580 tx=0x7fb0500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48cb7f812d132f8e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb041ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb041ffb640 1 -- 172.21.15.77:0/2676066680 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb050003d70 con 0x559badaeab40 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb054032db0 con 0x559badaeab40 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb041ffb640 1 -- 172.21.15.77:0/2676066680 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb050005020 con 0x559badaeab40 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb041ffb640 1 -- 172.21.15.77:0/2676066680 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb050005220 con 0x559badaeab40 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb06305c740 10 client.15201 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:42.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559bad7592d0 con 0x559badaeab40 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb041ffb640 1 -- 172.21.15.77:0/2676066680 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb054032db0 con 0x559badaeab40 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb041ffb640 1 -- 172.21.15.77:0/2676066680 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fb05001d040 con 0x559badaeab40 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb06305c740 10 client.15201 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb06305c740 10 client.15201 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.413+0000 7fb06305c740 20 client.15201 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.414+0000 7fb06305c740 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badafbbf0 0x559badafd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.414+0000 7fb06305c740 4 client.15201 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.414+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bada521c0 con 0x559badafbbf0 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.414+0000 7fb05e211640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badafbbf0 0x559badafd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.414+0000 7fb041ffb640 10 client.15201 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.414+0000 7fb05e211640 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badafbbf0 0x559badafd060 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d29fdfd3b1c3562c server_cookie=fe34bab2f6cb67a1 in_seq=0 out_seq=0 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb041ffb640 1 -- 172.21.15.77:0/2676066680 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559bada521c0 con 0x559badafbbf0 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb041ffb640 10 client.15201 handle_command_reply: tid=0 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 client.15201 shutdown 2024-03-16T09:41:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 msgr2=0x559badaf66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf66b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb050004580 tx=0x7fb0500046c0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 shutdown_connections 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf66b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaef670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badafbbf0 0x559badafd060 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 --2- 172.21.15.77:0/2676066680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae4fe0 0x559badaef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb041ffb640 0 client.15201 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 >> 172.21.15.77:0/2676066680 conn(0x559badadc140 msgr2=0x559badadb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 shutdown_connections 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 1 -- 172.21.15.77:0/2676066680 wait complete. 2024-03-16T09:41:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.415+0000 7fb06305c740 20 client.15201 trim_cache size 0 max 16384 2024-03-16T09:41:42.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:41 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2535892019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:41 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4100816061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:41 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1391646569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:42.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:41 smithi119 ceph-mon[35632]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 Processor -- start 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 -- start start 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae4fe0 0x559badaea180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaf6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badaeab40 0x559badadc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559badafb3a0 con 0x559badae4fe0 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559badafba70 con 0x559badaeab40 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb06305c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559badafc140 con 0x559badae59d0 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae4fe0 0x559badaea180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae4fe0 0x559badaea180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60002/0 (socket says 172.21.15.77:60002) 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 -- 172.21.15.77:0/3484749741 learned_addr learned my addr 172.21.15.77:0/3484749741 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05da10640 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaf6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05ea12640 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badaeab40 0x559badadc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 -- 172.21.15.77:0/3484749741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 msgr2=0x559badaf6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaf6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 -- 172.21.15.77:0/3484749741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badaeab40 msgr2=0x559badadc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badaeab40 0x559badadc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05e211640 1 -- 172.21.15.77:0/3484749741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bad8236b0 con 0x559badae4fe0 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.433+0000 7fb05ea12640 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badaeab40 0x559badadc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb05e211640 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae4fe0 0x559badaea180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb05402f020 tx=0x7fb054000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb67e9b0f8a387f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb043fff640 1 -- 172.21.15.77:0/3484749741 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054056070 con 0x559badae4fe0 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb043fff640 1 -- 172.21.15.77:0/3484749741 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0540325f0 con 0x559badae4fe0 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb043fff640 1 -- 172.21.15.77:0/3484749741 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054039040 con 0x559badae4fe0 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 -- 172.21.15.77:0/3484749741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae4fe0 msgr2=0x559badaea180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae4fe0 0x559badaea180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb05402f020 tx=0x7fb054000f40 comp rx=0 tx=0).stop 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 -- 172.21.15.77:0/3484749741 shutdown_connections 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badaeab40 0x559badadc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae59d0 0x559badaf6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 --2- 172.21.15.77:0/3484749741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badae4fe0 0x559badaea180 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 -- 172.21.15.77:0/3484749741 >> 172.21.15.77:0/3484749741 conn(0x559badadc140 msgr2=0x559badaeed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 -- 172.21.15.77:0/3484749741 shutdown_connections 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.434+0000 7fb06305c740 1 -- 172.21.15.77:0/3484749741 wait complete. 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 -- start start 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 Processor -- start 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae4fe0 0x559badaf0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae59d0 0x559badaf6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559badafcee0 con 0x559badaeab40 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559badafc810 con 0x559badae59d0 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559badafd5b0 con 0x559badae4fe0 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb06305c740 10 client.0 _ll_register_callbacks cb 0x7fb050004410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60004/0 (socket says 172.21.15.77:60004) 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 -- 172.21.15.77:0/3281626329 learned_addr learned my addr 172.21.15.77:0/3281626329 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05e211640 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae4fe0 0x559badaf0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05da10640 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae59d0 0x559badaf6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 -- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae4fe0 msgr2=0x559badaf0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae4fe0 0x559badaf0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 -- 172.21.15.77:0/3281626329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae59d0 msgr2=0x559badaf6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae59d0 0x559badaf6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 -- 172.21.15.77:0/3281626329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559badaf1a70 con 0x559badaeab40 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb05ea12640 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf1430 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb050009880 tx=0x7fb050005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fe1a6238f67cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb041ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb041ffb640 1 -- 172.21.15.77:0/3281626329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05001fb60 con 0x559badaeab40 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.435+0000 7fb041ffb640 1 -- 172.21.15.77:0/3281626329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb05001fd00 con 0x559badaeab40 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0540325f0 con 0x559badaeab40 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb041ffb640 1 -- 172.21.15.77:0/3281626329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb050045050 con 0x559badaeab40 2024-03-16T09:41:42.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 10 client.15207 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559badb60a60 con 0x559badaeab40 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb041ffb640 1 -- 172.21.15.77:0/3281626329 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0540325f0 con 0x559badaeab40 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb041ffb640 1 -- 172.21.15.77:0/3281626329 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fb05001d040 con 0x559badaeab40 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 10 client.15207 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 10 client.15207 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 20 client.15207 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badb656d0 0x559badb643e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 4 client.15207 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "status"}) -- 0x559bada4c510 con 0x559badb656d0 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb05e211640 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badb656d0 0x559badb643e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb041ffb640 10 client.15207 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.436+0000 7fb05e211640 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badb656d0 0x559badb643e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e3ff253577e1d80 server_cookie=121e0f189e6a3ad1 in_seq=0 out_seq=0 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb041ffb640 1 -- 172.21.15.77:0/3281626329 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559bada4c510 con 0x559badb656d0 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb041ffb640 10 client.15207 handle_command_reply: tid=0 2024-03-16T09:41:42.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 client.15207 shutdown 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 msgr2=0x559badaf1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf1430 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb050009880 tx=0x7fb050005700 comp rx=0 tx=0).stop 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 shutdown_connections 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559badaeab40 0x559badaf1430 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x559badb656d0 0x559badb643e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559badae59d0 0x559badaf6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 --2- 172.21.15.77:0/3281626329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559badae4fe0 0x559badaf0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb041ffb640 0 client.15207 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 >> 172.21.15.77:0/3281626329 conn(0x559badadc140 msgr2=0x559badaeed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 shutdown_connections 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 1 -- 172.21.15.77:0/3281626329 wait complete. 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb06305c740 20 client.15207 trim_cache size 0 max 16384 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:42.447 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 2, 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24481, 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 103.714500186, 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 20, 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 88, 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 86, 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 120.10334245599999 2024-03-16T09:41:42.448 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0440a2880 msgr2=0x7fb0440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0440a2880 0x7fb0440a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x559badae6230 tx=0x7fb050005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581018d0 msgr2=0x7fb05819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581018d0 0x7fb05819c810 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb04800e9f0 tx=0x7fb04800edf0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 shutdown_connections 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0440a2880 0x7fb0440a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb058102290 0x7fb05819ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0581018d0 0x7fb05819c810 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 --2- 172.21.15.77:0/1905283702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0581006e0 0x7fb0580694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.437+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 >> 172.21.15.77:0/1905283702 conn(0x7fb0580fbc70 msgr2=0x7fb058104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.438+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 shutdown_connections 2024-03-16T09:41:42.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.438+0000 7fb060475640 1 -- 172.21.15.77:0/1905283702 wait complete. 2024-03-16T09:41:42.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T09:41:42.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 -- 172.21.15.77:0/816212942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101006e0 msgr2=0x7fbe10100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 --2- 172.21.15.77:0/816212942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101006e0 0x7fbe10100b00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009ca0 tx=0x7fbe0802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 -- 172.21.15.77:0/816212942 shutdown_connections 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 --2- 172.21.15.77:0/816212942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe10102290 0x7fbe10108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 --2- 172.21.15.77:0/816212942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe101018d0 0x7fbe10101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 --2- 172.21.15.77:0/816212942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101006e0 0x7fbe10100b00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 -- 172.21.15.77:0/816212942 >> 172.21.15.77:0/816212942 conn(0x7fbe100fbc70 msgr2=0x7fbe100fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 -- 172.21.15.77:0/816212942 shutdown_connections 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 -- 172.21.15.77:0/816212942 wait complete. 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 Processor -- start 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.550+0000 7fbe17709640 1 -- start start 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe101006e0 0x7fbe10197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101018d0 0x7fbe10197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe10102290 0x7fbe1019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1010c1e0 con 0x7fbe101018d0 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1010c060 con 0x7fbe10102290 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1010c360 con 0x7fbe101006e0 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101018d0 0x7fbe10197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101018d0 0x7fbe10197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60018/0 (socket says 172.21.15.77:60018) 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3191233800 learned_addr learned my addr 172.21.15.77:0/3191233800 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe10102290 0x7fbe1019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe101006e0 0x7fbe10197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3191233800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe101006e0 msgr2=0x7fbe10197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe101006e0 0x7fbe10197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3191233800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe10102290 msgr2=0x7fbe1019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe10102290 0x7fbe1019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3191233800 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1019f6c0 con 0x7fbe101018d0 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe101006e0 0x7fbe10197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe14ca4640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101018d0 0x7fbe10197fe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400ea60 tx=0x7fbe0400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/3191233800 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0400cd20 con 0x7fbe101018d0 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/3191233800 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe040035f0 con 0x7fbe101018d0 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/3191233800 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04010690 con 0x7fbe101018d0 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1019f9b0 con 0x7fbe101018d0 2024-03-16T09:41:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.551+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe1019ff50 con 0x7fbe101018d0 2024-03-16T09:41:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.553+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/3191233800 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbe040037a0 con 0x7fbe101018d0 2024-03-16T09:41:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 Processor -- start 2024-03-16T09:41:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 -- start start 2024-03-16T09:41:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b011eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b01240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0126bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5593b0082c80 con 0x5593b0124730 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5593b002baa0 con 0x5593b011f5c0 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593b007ed20 con 0x5593b011ebd0 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe0e7fc640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbdf80a2880 0x7fbdf80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.554+0000 7fbe14ca4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0126bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe154a5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b01240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe14ca4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0126bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60020/0 (socket says 172.21.15.77:60020) 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe14ca4640 1 -- 172.21.15.77:0/1433068628 learned_addr learned my addr 172.21.15.77:0/1433068628 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b011eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbdf80a2880 0x7fbdf80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe154a5640 1 -- 172.21.15.77:0/1433068628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 msgr2=0x5593b011eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe154a5640 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b011eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe154a5640 1 -- 172.21.15.77:0/1433068628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 msgr2=0x5593b0126bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe154a5640 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0126bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/3191233800 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbe040cedc0 con 0x7fbe101018d0 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe154a5640 1 -- 172.21.15.77:0/1433068628 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593b004dff0 con 0x5593b011f5c0 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbdf80a2880 0x7fbdf80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x5593b011fe20 tx=0x7fbe00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbe154a5640 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b01240c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbe0802c630 tx=0x7fbe08009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c27ecb8d1ad355c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbdf77fe640 1 -- 172.21.15.77:0/1433068628 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08043020 con 0x5593b011f5c0 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbdf77fe640 1 -- 172.21.15.77:0/1433068628 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe08032e70 con 0x5593b011f5c0 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.555+0000 7fbdf77fe640 1 -- 172.21.15.77:0/1433068628 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08039040 con 0x5593b011f5c0 2024-03-16T09:41:42.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1433068628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 msgr2=0x5593b01240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b01240c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbe0802c630 tx=0x7fbe08009880 comp rx=0 tx=0).stop 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1433068628 shutdown_connections 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0126bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b01240c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1433068628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b011eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1433068628 >> 172.21.15.77:0/1433068628 conn(0x5593b011a140 msgr2=0x5593b011a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1433068628 shutdown_connections 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1433068628 wait complete. 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 -- start start 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.556+0000 7fbe1a3d1740 1 Processor -- start 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b0128d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b011f5c0 0x5593b01292c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b0124730 0x5593b0130300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5593b0127680 con 0x5593b011f5c0 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5593b003eef0 con 0x5593b0124730 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593b0127800 con 0x5593b011ebd0 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 10 client.0 _ll_register_callbacks cb 0x5593b0039dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b011f5c0 0x5593b01292c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b011f5c0 0x5593b01292c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60024/0 (socket says 172.21.15.77:60024) 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3884926134 learned_addr learned my addr 172.21.15.77:0/3884926134 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b0124730 0x5593b0130300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b0128d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3884926134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 msgr2=0x5593b0128d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b0128d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b0124730 msgr2=0x5593b0130300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b0124730 0x5593b0130300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 -- 172.21.15.77:0/3884926134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593b004ba30 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe14ca4640 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b011f5c0 0x5593b01292c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009230 tx=0x7fbe0409b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80595baad3c014a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbdf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3884926134 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04003b30 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3884926134 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbe040cc980 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5593b00901c0 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3884926134 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fbe0409be10 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.557+0000 7fbe1a3d1740 10 client.15219 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5593afd972d0 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3884926134 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5593b00901c0 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3884926134 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fbe0409ab60 con 0x5593b011f5c0 2024-03-16T09:41:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe1a3d1740 10 client.15219 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe1a3d1740 10 client.15219 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe1a3d1740 20 client.15219 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b0136c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe1a3d1740 4 client.15219 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593b008a510 con 0x5593b0135800 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b0136c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b0136c70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c66fc55e8e6583d server_cookie=97908ef848f4e458 in_seq=0 out_seq=0 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.558+0000 7fbdf4ff9640 10 client.15219 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3884926134 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593b008a510 con 0x5593b0135800 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbdf4ff9640 10 client.15219 handle_command_reply: tid=0 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 client.15219 shutdown 2024-03-16T09:41:42.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b011f5c0 msgr2=0x5593b01292c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b011f5c0 0x5593b01292c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009230 tx=0x7fbe0409b070 comp rx=0 tx=0).stop 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 shutdown_connections 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b0136c70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbdf4ff9640 0 client.15219 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b0124730 0x5593b0130300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b011f5c0 0x5593b01292c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3884926134 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b0128d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 >> 172.21.15.77:0/3884926134 conn(0x5593b011a140 msgr2=0x5593b0119380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 shutdown_connections 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3884926134 wait complete. 2024-03-16T09:41:42.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.559+0000 7fbe1a3d1740 20 client.15219 trim_cache size 0 max 16384 2024-03-16T09:41:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 Processor -- start 2024-03-16T09:41:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 -- start start 2024-03-16T09:41:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011ebd0 0x5593b011a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011f5c0 0x5593b01302c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0123e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5593b0134fb0 con 0x5593b0124730 2024-03-16T09:41:42.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5593b0135680 con 0x5593b011ebd0 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593b0135d50 con 0x5593b011f5c0 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.577+0000 7fbe15ca6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0123e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0123e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60034/0 (socket says 172.21.15.77:60034) 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 -- 172.21.15.77:0/1924111547 learned_addr learned my addr 172.21.15.77:0/1924111547 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe154a5640 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011ebd0 0x5593b011a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 -- 172.21.15.77:0/1924111547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011f5c0 msgr2=0x5593b01302c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011f5c0 0x5593b01302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 -- 172.21.15.77:0/1924111547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011ebd0 msgr2=0x5593b011a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011ebd0 0x5593b011a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 -- 172.21.15.77:0/1924111547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593afe616b0 con 0x5593b0124730 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0123e10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbe00004060 tx=0x7fbe00008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb96cb06637ed795 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbdf6ffd640 1 -- 172.21.15.77:0/1924111547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0002f070 con 0x5593b0124730 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbdf6ffd640 1 -- 172.21.15.77:0/1924111547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe0000fe70 con 0x5593b0124730 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.578+0000 7fbdf6ffd640 1 -- 172.21.15.77:0/1924111547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0002ab30 con 0x5593b0124730 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1924111547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 msgr2=0x5593b0123e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0123e10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbe00004060 tx=0x7fbe00008000 comp rx=0 tx=0).stop 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1924111547 shutdown_connections 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b0123e10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011f5c0 0x5593b01302c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/1924111547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011ebd0 0x5593b011a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1924111547 >> 172.21.15.77:0/1924111547 conn(0x5593b011a140 msgr2=0x5593b011dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1924111547 shutdown_connections 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/1924111547 wait complete. 2024-03-16T09:41:42.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- start start 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 Processor -- start 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b012bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b012c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b012f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5593b0136af0 con 0x5593b0124730 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5593b0136420 con 0x5593b011f5c0 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593b01371c0 con 0x5593b011ebd0 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe1a3d1740 10 client.0 _ll_register_callbacks cb 0x7fbe04003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.579+0000 7fbe15ca6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b012f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b012f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60036/0 (socket says 172.21.15.77:60036) 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 -- 172.21.15.77:0/3553040691 learned_addr learned my addr 172.21.15.77:0/3553040691 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe14ca4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b012c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b012bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 -- 172.21.15.77:0/3553040691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 msgr2=0x5593b012bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b012bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 -- 172.21.15.77:0/3553040691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 msgr2=0x5593b012c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b012c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 -- 172.21.15.77:0/3553040691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593b012f680 con 0x5593b0124730 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe15ca6640 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b012f090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009700 tx=0x7fbe00005510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80154ede6d86b518 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbdf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3553040691 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00037020 con 0x5593b0124730 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3553040691 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbe0002aa30 con 0x5593b0124730 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe0000fe70 con 0x5593b0124730 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3553040691 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fbe0002f040 con 0x5593b0124730 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe1a3d1740 10 client.15225 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:42.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5593b019b660 con 0x5593b0124730 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3553040691 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe0000fe70 con 0x5593b0124730 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.580+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3553040691 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fbe00032030 con 0x5593b0124730 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 10 client.15225 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 10 client.15225 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 20 client.15225 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 4 client.15225 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "status"}) -- 0x5593b0008a00 con 0x5593b0135800 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe154a5640 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b019eba0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d94f23164ff0f75 server_cookie=2446c312b249d24 in_seq=0 out_seq=0 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbdf4ff9640 10 client.15225 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:42.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbdf4ff9640 1 -- 172.21.15.77:0/3553040691 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5593b0008a00 con 0x5593b0135800 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbdf4ff9640 10 client.15225 handle_command_reply: tid=0 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 client.15225 shutdown 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 msgr2=0x5593b012f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b012f090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009700 tx=0x7fbe00005510 comp rx=0 tx=0).stop 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 shutdown_connections 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5593b0135800 0x5593b019eba0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593b0124730 0x5593b012f090 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593b011f5c0 0x5593b012c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbdf4ff9640 0 client.15225 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 --2- 172.21.15.77:0/3553040691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593b011ebd0 0x5593b012bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.581+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 >> 172.21.15.77:0/3553040691 conn(0x5593b011a140 msgr2=0x5593b011dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 shutdown_connections 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe1a3d1740 1 -- 172.21.15.77:0/3553040691 wait complete. 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe1a3d1740 20 client.15225 trim_cache size 0 max 16384 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 1, 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24503, 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T09:41:42.593 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T09:41:42.594 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 104.87877592300001, 2024-03-16T09:41:42.594 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 20, 2024-03-16T09:41:42.594 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 88, 2024-03-16T09:41:42.594 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 86, 2024-03-16T09:41:42.594 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 122.882501495 2024-03-16T09:41:42.594 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbdf80a2880 msgr2=0x7fbdf80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbdf80a2880 0x7fbdf80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x5593b011fe20 tx=0x7fbe00005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101018d0 msgr2=0x7fbe10197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101018d0 0x7fbe10197fe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400ea60 tx=0x7fbe0400ee60 comp rx=0 tx=0).stop 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 shutdown_connections 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbdf80a2880 0x7fbdf80a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe10102290 0x7fbe1019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe101018d0 0x7fbe10197fe0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 --2- 172.21.15.77:0/3191233800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe101006e0 0x7fbe10197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 >> 172.21.15.77:0/3191233800 conn(0x7fbe100fbc70 msgr2=0x7fbe10068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 shutdown_connections 2024-03-16T09:41:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.582+0000 7fbe17709640 1 -- 172.21.15.77:0/3191233800 wait complete. 2024-03-16T09:41:42.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T09:41:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 -- 172.21.15.77:0/1553646094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 msgr2=0x7f1d64109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 --2- 172.21.15.77:0/1553646094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d64109c30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c009ca0 tx=0x7f1d5c02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 -- 172.21.15.77:0/1553646094 shutdown_connections 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 --2- 172.21.15.77:0/1553646094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d64109c30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 --2- 172.21.15.77:0/1553646094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d641029e0 0x7f1d64102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 --2- 172.21.15.77:0/1553646094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d641017f0 0x7f1d64101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 -- 172.21.15.77:0/1553646094 >> 172.21.15.77:0/1553646094 conn(0x7f1d640fcd80 msgr2=0x7f1d640ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 -- 172.21.15.77:0/1553646094 shutdown_connections 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.696+0000 7f1d6a459640 1 -- 172.21.15.77:0/1553646094 wait complete. 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 Processor -- start 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 -- start start 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d641017f0 0x7f1d641979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d641029e0 0x7f1d64197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d6419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6410d320 con 0x7f1d641033a0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6410d1a0 con 0x7f1d641017f0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d6a459640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6410d4a0 con 0x7f1d641029e0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d689f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d6419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d689f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d6419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60054/0 (socket says 172.21.15.77:60054) 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.697+0000 7f1d689f6640 1 -- 172.21.15.77:0/358863672 learned_addr learned my addr 172.21.15.77:0/358863672 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d63fff640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d641017f0 0x7f1d641979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d637fe640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d641029e0 0x7f1d64197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d689f6640 1 -- 172.21.15.77:0/358863672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d641029e0 msgr2=0x7f1d64197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d689f6640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d641029e0 0x7f1d64197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d689f6640 1 -- 172.21.15.77:0/358863672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d641017f0 msgr2=0x7f1d641979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d689f6640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d641017f0 0x7f1d641979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d689f6640 1 -- 172.21.15.77:0/358863672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6419f610 con 0x7f1d641033a0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d689f6640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d6419efa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c02c700 tx=0x7f1d5c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d617fa640 1 -- 172.21.15.77:0/358863672 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c041020 con 0x7f1d641033a0 2024-03-16T09:41:42.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d617fa640 1 -- 172.21.15.77:0/358863672 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d5c003e20 con 0x7f1d641033a0 2024-03-16T09:41:42.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d6419f8a0 con 0x7f1d641033a0 2024-03-16T09:41:42.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d617fa640 1 -- 172.21.15.77:0/358863672 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c039040 con 0x7f1d641033a0 2024-03-16T09:41:42.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.698+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d6419fd90 con 0x7f1d641033a0 2024-03-16T09:41:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.700+0000 7f1d617fa640 1 -- 172.21.15.77:0/358863672 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1d5c044020 con 0x7f1d641033a0 2024-03-16T09:41:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 Processor -- start 2024-03-16T09:41:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 -- start start 2024-03-16T09:41:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d617fa640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1d4c0a2800 0x7f1d4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a02678aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a0267900c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026792bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0266eec80 con 0x55a02678b5c0 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a026697aa0 con 0x55a02678abd0 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d6d043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a0266ead20 con 0x55a026790730 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d637fe640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1d4c0a2800 0x7f1d4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a02678aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a02678aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:49990/0 (socket says 172.21.15.77:49990) 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 -- 172.21.15.77:0/1598795676 learned_addr learned my addr 172.21.15.77:0/1598795676 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d689f6640 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026792bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d637fe640 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a0267900c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d617fa640 1 -- 172.21.15.77:0/358863672 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1d5c0f0be0 con 0x7f1d641033a0 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 -- 172.21.15.77:0/1598795676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 msgr2=0x55a026792bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026792bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 -- 172.21.15.77:0/1598795676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 msgr2=0x55a0267900c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a0267900c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 -- 172.21.15.77:0/1598795676 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0266b9ff0 con 0x55a02678abd0 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d637fe640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1d4c0a2800 0x7f1d4c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d58009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.701+0000 7f1d63fff640 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a02678aff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1d54009c80 tx=0x7f1d54011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1caec7c4babfed05 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d4a7fc640 1 -- 172.21.15.77:0/1598795676 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5401e070 con 0x55a02678abd0 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d4a7fc640 1 -- 172.21.15.77:0/1598795676 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d540198e0 con 0x55a02678abd0 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d4a7fc640 1 -- 172.21.15.77:0/1598795676 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400e820 con 0x55a02678abd0 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 -- 172.21.15.77:0/1598795676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 msgr2=0x55a02678aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a02678aff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1d54009c80 tx=0x7f1d54011810 comp rx=0 tx=0).stop 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 -- 172.21.15.77:0/1598795676 shutdown_connections 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026792bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a0267900c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 --2- 172.21.15.77:0/1598795676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a02678aff0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 -- 172.21.15.77:0/1598795676 >> 172.21.15.77:0/1598795676 conn(0x55a026786140 msgr2=0x55a026786540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 -- 172.21.15.77:0/1598795676 shutdown_connections 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.702+0000 7f1d6d043740 1 -- 172.21.15.77:0/1598795676 wait complete. 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 -- start start 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 Processor -- start 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026795380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a02679c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0267936e0 con 0x55a02678b5c0 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a0266aaef0 con 0x55a026790730 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a026793860 con 0x55a02678abd0 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d6d043740 10 client.0 _ll_register_callbacks cb 0x55a0266a5dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026795380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026795380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60066/0 (socket says 172.21.15.77:60066) 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d689f6640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a02679c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d637fe640 1 -- 172.21.15.77:0/2956645420 learned_addr learned my addr 172.21.15.77:0/2956645420 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.703+0000 7f1d63fff640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d637fe640 1 -- 172.21.15.77:0/2956645420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 msgr2=0x55a026794e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d637fe640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d637fe640 1 -- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 msgr2=0x55a02679c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d637fe640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a02679c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d637fe640 1 -- 172.21.15.77:0/2956645420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0266b7a30 con 0x55a02678b5c0 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d689f6640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a02679c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d637fe640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026795380 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009f60 tx=0x7f1d5801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6111fc414349d99a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d437fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:42.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d437fe640 1 -- 172.21.15.77:0/2956645420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d580044c0 con 0x55a02678b5c0 2024-03-16T09:41:42.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d437fe640 1 -- 172.21.15.77:0/2956645420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1d58003800 con 0x55a02678b5c0 2024-03-16T09:41:42.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d540198e0 con 0x55a02678b5c0 2024-03-16T09:41:42.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d437fe640 1 -- 172.21.15.77:0/2956645420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1d58003a00 con 0x55a02678b5c0 2024-03-16T09:41:42.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 10 client.15237 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a0264032d0 con 0x55a02678b5c0 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d437fe640 1 -- 172.21.15.77:0/2956645420 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d540198e0 con 0x55a02678b5c0 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d437fe640 1 -- 172.21.15.77:0/2956645420 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f1d5801fb10 con 0x55a02678b5c0 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 10 client.15237 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 10 client.15237 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 20 client.15237 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0267a1900 0x55a0267a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 4 client.15237 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.704+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0266fc1c0 con 0x55a0267a1900 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.705+0000 7f1d63fff640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0267a1900 0x55a0267a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.705+0000 7f1d437fe640 10 client.15237 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.705+0000 7f1d63fff640 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0267a1900 0x55a0267a2d70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa946894e06e4d3d server_cookie=6764b6f8ecc00147 in_seq=0 out_seq=0 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d437fe640 1 -- 172.21.15.77:0/2956645420 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a0266fc1c0 con 0x55a0267a1900 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d437fe640 10 client.15237 handle_command_reply: tid=0 2024-03-16T09:41:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 client.15237 shutdown 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 msgr2=0x55a026795380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026795380 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009f60 tx=0x7f1d5801d000 comp rx=0 tx=0).stop 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 shutdown_connections 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0267a1900 0x55a0267a2d70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a02679c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d437fe640 0 client.15237 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026795380 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 --2- 172.21.15.77:0/2956645420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 >> 172.21.15.77:0/2956645420 conn(0x55a026786140 msgr2=0x55a026785380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 shutdown_connections 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 1 -- 172.21.15.77:0/2956645420 wait complete. 2024-03-16T09:41:42.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.706+0000 7f1d6d043740 20 client.15237 trim_cache size 0 max 16384 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 Processor -- start 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 -- start start 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a02679c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026795380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0267a10b0 con 0x55a02678b5c0 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a0267a1780 con 0x55a02678abd0 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d6d043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a0267a1e50 con 0x55a026790730 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a02679c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a02679c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60076/0 (socket says 172.21.15.77:60076) 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.724+0000 7f1d637fe640 1 -- 172.21.15.77:0/4045667629 learned_addr learned my addr 172.21.15.77:0/4045667629 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d63fff640 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d689f6640 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026795380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d637fe640 1 -- 172.21.15.77:0/4045667629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 msgr2=0x55a026795380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d637fe640 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026795380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d637fe640 1 -- 172.21.15.77:0/4045667629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 msgr2=0x55a026794e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d637fe640 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d637fe640 1 -- 172.21.15.77:0/4045667629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0264cd6b0 con 0x55a02678b5c0 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d689f6640 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026795380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d63fff640 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d637fe640 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a02679c310 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800f7b0 tx=0x7f1d580057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b4b1aab126eaef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d49ffb640 1 -- 172.21.15.77:0/4045667629 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58038040 con 0x55a02678b5c0 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d49ffb640 1 -- 172.21.15.77:0/4045667629 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d5802e030 con 0x55a02678b5c0 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.725+0000 7f1d49ffb640 1 -- 172.21.15.77:0/4045667629 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58010550 con 0x55a02678b5c0 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- 172.21.15.77:0/4045667629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 msgr2=0x55a02679c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a02679c310 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800f7b0 tx=0x7f1d580057d0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- 172.21.15.77:0/4045667629 shutdown_connections 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a026790730 0x55a026795380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a02679c310 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 --2- 172.21.15.77:0/4045667629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a02678abd0 0x55a026794e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- 172.21.15.77:0/4045667629 >> 172.21.15.77:0/4045667629 conn(0x55a026786140 msgr2=0x55a02678fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- 172.21.15.77:0/4045667629 shutdown_connections 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- 172.21.15.77:0/4045667629 wait complete. 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- start start 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 Processor -- start 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a02679a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026803650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a026805ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0267a2bf0 con 0x55a02678b5c0 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a0267a2520 con 0x55a026790730 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a0267a32c0 con 0x55a02678abd0 2024-03-16T09:41:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d6d043740 10 client.0 _ll_register_callbacks cb 0x7f1d58009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026803650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026803650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60088/0 (socket says 172.21.15.77:60088) 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d637fe640 1 -- 172.21.15.77:0/32907819 learned_addr learned my addr 172.21.15.77:0/32907819 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.726+0000 7f1d63fff640 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a02679a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d689f6640 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a026805ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d637fe640 1 -- 172.21.15.77:0/32907819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 msgr2=0x55a02679a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d637fe640 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a02679a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d637fe640 1 -- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 msgr2=0x55a026805ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d637fe640 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a026805ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d637fe640 1 -- 172.21.15.77:0/32907819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0268060e0 con 0x55a02678b5c0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d637fe640 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026803650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800fe60 tx=0x7f1d580098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3611a612ab1549 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d437fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d437fe640 1 -- 172.21.15.77:0/32907819 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58038070 con 0x55a02678b5c0 2024-03-16T09:41:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d437fe640 1 -- 172.21.15.77:0/32907819 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1d58003ba0 con 0x55a02678b5c0 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d5802e030 con 0x55a02678b5c0 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d437fe640 1 -- 172.21.15.77:0/32907819 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1d58011670 con 0x55a02678b5c0 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d6d043740 10 client.15243 fetch_fsmap learned FSMap version 20 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a026807340 con 0x55a02678b5c0 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d437fe640 1 -- 172.21.15.77:0/32907819 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d5802e030 con 0x55a02678b5c0 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d437fe640 1 -- 172.21.15.77:0/32907819 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f1d580105f0 con 0x55a02678b5c0 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.727+0000 7f1d6d043740 10 client.15243 fetch_fsmap finished waiting for FSMap version 20 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 10 client.15243 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 20 client.15243 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a02680bd10 0x55a02680aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 4 client.15243 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "status"}) -- 0x55a0266f6510 con 0x55a02680bd10 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d63fff640 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a02680bd10 0x55a02680aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d437fe640 10 client.15243 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d63fff640 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a02680bd10 0x55a02680aa20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f50f6975b982640 server_cookie=bb387c0385cc70be in_seq=0 out_seq=0 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d437fe640 1 -- 172.21.15.77:0/32907819 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55a0266f6510 con 0x55a02680bd10 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d437fe640 10 client.15243 handle_command_reply: tid=0 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 client.15243 shutdown 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 msgr2=0x55a026803650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026803650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800fe60 tx=0x7f1d580098d0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 shutdown_connections 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a02680bd10 0x55a02680aa20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a026790730 0x55a026805ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a02678b5c0 0x55a026803650 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 --2- 172.21.15.77:0/32907819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a02678abd0 0x55a02679a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d437fe640 0 client.15243 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.728+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 >> 172.21.15.77:0/32907819 conn(0x55a026786140 msgr2=0x55a026794e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 shutdown_connections 2024-03-16T09:41:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6d043740 1 -- 172.21.15.77:0/32907819 wait complete. 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6d043740 20 client.15243 trim_cache size 0 max 16384 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "e287e1d6-e377-11ee-95c9-87774f69a715", 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24545, 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 106.550637791, 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 20, 2024-03-16T09:41:42.740 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 88, 2024-03-16T09:41:42.741 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 86, 2024-03-16T09:41:42.741 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 112.124851429 2024-03-16T09:41:42.741 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1d4c0a2800 msgr2=0x7f1d4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1d4c0a2800 0x7f1d4c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d58009b80 comp rx=0 tx=0).stop 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 msgr2=0x7f1d6419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d6419efa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c02c700 tx=0x7f1d5c0328a0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 shutdown_connections 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f1d4c0a2800 0x7f1d4c0a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d641033a0 0x7f1d6419efa0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d641029e0 0x7f1d64197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 --2- 172.21.15.77:0/358863672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d641017f0 0x7f1d641979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 >> 172.21.15.77:0/358863672 conn(0x7f1d640fcd80 msgr2=0x7f1d64105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 shutdown_connections 2024-03-16T09:41:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:42.729+0000 7f1d6a459640 1 -- 172.21.15.77:0/358863672 wait complete. 2024-03-16T09:41:42.743 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-16T09:41:42.767 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-16T09:41:42.767 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-16T09:41:42.767 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-16T09:41:42.767 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T09:41:42.767 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:42.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:42.768 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-16T09:41:42.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:42.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-03-16T09:41:42.851 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-03-16T09:41:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:41:42.853 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-16T09:41:42.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T09:41:43.136 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:41:43.201 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:42 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3736276752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:43.202 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:42 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2025065647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:43.257 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:42 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3736276752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:43.257 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:42 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2025065647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:43.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:42 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3736276752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:43.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:42 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2025065647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:44.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:43 smithi077 ceph-mon[32171]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:44.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:43 smithi129 ceph-mon[35841]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:44.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:43 smithi119 ceph-mon[35632]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:41:44.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 -- 172.21.15.77:0/1136404492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff0074500 msgr2=0x7efff00748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:44.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 --2- 172.21.15.77:0/1136404492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff0074500 0x7efff00748e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7effe0009990 tx=0x7effe0009ca0 comp rx=0 tx=0).stop 2024-03-16T09:41:44.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 -- 172.21.15.77:0/1136404492 shutdown_connections 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 --2- 172.21.15.77:0/1136404492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff01134a0 0x7efff0115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 --2- 172.21.15.77:0/1136404492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff0074500 0x7efff00748e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 --2- 172.21.15.77:0/1136404492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efff0074e20 0x7efff0112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 -- 172.21.15.77:0/1136404492 >> 172.21.15.77:0/1136404492 conn(0x7efff0079400 msgr2=0x7efff0079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 -- 172.21.15.77:0/1136404492 shutdown_connections 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 -- 172.21.15.77:0/1136404492 wait complete. 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 Processor -- start 2024-03-16T09:41:44.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.833+0000 7efff667c700 1 -- start start 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff667c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0074500 0x7efff019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff667c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efff0074e20 0x7efff019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff667c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff01134a0 0x7efff01a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff667c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efff0118b30 con 0x7efff01134a0 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff667c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efff01189b0 con 0x7efff0074e20 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff667c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efff0118cb0 con 0x7efff0074500 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff4c19700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff01134a0 0x7efff01a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7effeffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0074500 0x7efff019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7effeffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0074500 0x7efff019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:53614/0 (socket says 172.21.15.77:53614) 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff4c19700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff01134a0 0x7efff01a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60096/0 (socket says 172.21.15.77:60096) 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7effeffff700 1 -- 172.21.15.77:0/701965672 learned_addr learned my addr 172.21.15.77:0/701965672 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:44.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7effef7fe700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efff0074e20 0x7efff019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff4c19700 1 -- 172.21.15.77:0/701965672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0074500 msgr2=0x7efff019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff4c19700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0074500 0x7efff019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff4c19700 1 -- 172.21.15.77:0/701965672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efff0074e20 msgr2=0x7efff019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff4c19700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efff0074e20 0x7efff019f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7efff4c19700 1 -- 172.21.15.77:0/701965672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff01a4090 con 0x7efff01134a0 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7effef7fe700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efff0074e20 0x7efff019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.834+0000 7effeffff700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0074500 0x7efff019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.835+0000 7efff4c19700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff01134a0 0x7efff01a3950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7effe800ef90 tx=0x7effe800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.835+0000 7effed7fa700 1 -- 172.21.15.77:0/701965672 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe80092e0 con 0x7efff01134a0 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.835+0000 7effed7fa700 1 -- 172.21.15.77:0/701965672 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effe8004bb0 con 0x7efff01134a0 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.835+0000 7effed7fa700 1 -- 172.21.15.77:0/701965672 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8004020 con 0x7efff01134a0 2024-03-16T09:41:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.835+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff01a4320 con 0x7efff01134a0 2024-03-16T09:41:44.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.835+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff01abc00 con 0x7efff01134a0 2024-03-16T09:41:44.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.836+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff0068f90 con 0x7efff01134a0 2024-03-16T09:41:44.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.837+0000 7effed7fa700 1 -- 172.21.15.77:0/701965672 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7effe801b030 con 0x7efff01134a0 2024-03-16T09:41:44.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.838+0000 7effed7fa700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effd80a2910 0x7effd80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:44.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.838+0000 7effeffff700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effd80a2910 0x7effd80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:44.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.838+0000 7effed7fa700 1 -- 172.21.15.77:0/701965672 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7effe80ceea0 con 0x7efff01134a0 2024-03-16T09:41:44.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.838+0000 7effeffff700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effd80a2910 0x7effd80a4dd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7effe0006010 tx=0x7effe001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:44.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:44.842+0000 7effed7fa700 1 -- 172.21.15.77:0/701965672 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7effe8022020 con 0x7efff01134a0 2024-03-16T09:41:45.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.084+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efff0113190 con 0x7efff01134a0 2024-03-16T09:41:45.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.085+0000 7effed7fa700 1 -- 172.21.15.77:0/701965672 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7efff0113190 con 0x7efff01134a0 2024-03-16T09:41:45.088 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-16T09:41:45.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effd80a2910 msgr2=0x7effd80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effd80a2910 0x7effd80a4dd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7effe0006010 tx=0x7effe001a040 comp rx=0 tx=0).stop 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff01134a0 msgr2=0x7efff01a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff01134a0 0x7efff01a3950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7effe800ef90 tx=0x7effe800c3a0 comp rx=0 tx=0).stop 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 shutdown_connections 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effd80a2910 0x7effd80a4dd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efff01134a0 0x7efff01a3950 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0074500 0x7efff019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 --2- 172.21.15.77:0/701965672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efff0074e20 0x7efff019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 >> 172.21.15.77:0/701965672 conn(0x7efff0079400 msgr2=0x7efff01102d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 shutdown_connections 2024-03-16T09:41:45.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45.087+0000 7efff667c700 1 -- 172.21.15.77:0/701965672 wait complete. 2024-03-16T09:41:45.379 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:45 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/701965672' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T09:41:45.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:45 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/701965672' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T09:41:45.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:45 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/701965672' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T09:41:45.835 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-16T09:41:45.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:45.836 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-16T09:41:45.851 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T09:41:45.851 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T09:41:45.851 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42086sec preferred_lft 42086sec 2024-03-16T09:41:45.852 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-16T09:41:45.853 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-16T09:41:45.853 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T09:41:45.853 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T09:41:45.853 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T09:41:45.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-03-16T09:41:45.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-03-16T09:41:45.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-03-16T09:41:45.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T09:41:45.853 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T09:41:45.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T09:41:46.049 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T09:41:46.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:46.055 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T09:41:46.083 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-16T09:41:46.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:46.085 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-16T09:41:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-16T09:41:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T09:41:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-16T09:41:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-16T09:41:46.165 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T09:41:46.166 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T09:41:46.166 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T09:41:46.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-16T09:41:46.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-16T09:41:46.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-16T09:41:46.166 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T09:41:46.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T09:41:46.346 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:46 smithi077 ceph-mon[32171]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:46.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:46 smithi119 ceph-mon[35632]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:46.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T09:41:46.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:46.502 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-16T09:41:46.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:46.518 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-03-16T09:41:46.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:46 smithi129 ceph-mon[35841]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:41:46.572 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T09:41:46.573 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T09:41:46.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:41:46.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-16T09:41:46.573 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T09:41:46.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T09:41:46.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T09:41:46.694 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-16T09:41:46.695 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T09:41:46.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T09:41:46.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T09:41:46.836 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T09:41:46.836 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T09:41:46.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-16T09:41:46.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T09:41:46.837 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T09:41:46.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T09:41:46.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T09:41:46.919 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-16T09:41:46.919 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T09:41:46.919 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:46.933 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-16T09:41:46.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T09:41:46.933 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:46.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-16T09:41:47.055 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T09:41:47.111 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-16T09:41:47.111 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-16T09:41:47.111 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-16T09:41:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/906e5dbc089b0a3116389086de52a6621feecce3a9fe9ff27891d19b58cd7a16/merged 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/1ef34430c0ce9d442329192462963be97d3df4fa4c57a0a60cd6a1fad936d13d/merged 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/0 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T09:41:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/41a6415788e4c303ad62c44f76fdf2f33cf856e7a8b3fb46e0317e8fa5f92bdb/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/236809ad60caee1cddc5b4db6dae2f85c45c13ef6679528058fc24f8dc0a7a21/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/dbe0827f342ef04efe291b0ae1bfb60ea9c97e38cfa24c319d4a54d84f46e572/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/85efabf09ed948bea2ffcc3baeed72e21f2310045b9d1b6a92bd3d170d0bbcd1/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/bff085ccab23f994d13295f2d9cc3e5a8816cadd33d79b5610352c5e5c6ae535/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/9a49fa8d68f1800b095c0e1dcb3927d0b2d488fb44e59a0dd31bcdb021afcc71/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/7051c118f8c906d7832c257e0ac0b17a4be0b8cd75e2cc12df3ce3f2465c92d6/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/05bbac6ee220154c57896c3787335fdd8b436394a905639777c6965d6dd70e88/merged 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:41:47.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:47.115 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-16T09:41:47.166 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-16T09:41:47.166 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/d6cb7cbe-a826-46f6-811e-8b60ec9c6a86 --client_fs=cephfs) 2024-03-16T09:41:47.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-16T09:41:47.235 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T09:41:47.291 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-16T09:41:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/906e5dbc089b0a3116389086de52a6621feecce3a9fe9ff27891d19b58cd7a16/merged 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/1ef34430c0ce9d442329192462963be97d3df4fa4c57a0a60cd6a1fad936d13d/merged 2024-03-16T09:41:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/0 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/41a6415788e4c303ad62c44f76fdf2f33cf856e7a8b3fb46e0317e8fa5f92bdb/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/236809ad60caee1cddc5b4db6dae2f85c45c13ef6679528058fc24f8dc0a7a21/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/dbe0827f342ef04efe291b0ae1bfb60ea9c97e38cfa24c319d4a54d84f46e572/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/85efabf09ed948bea2ffcc3baeed72e21f2310045b9d1b6a92bd3d170d0bbcd1/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/bff085ccab23f994d13295f2d9cc3e5a8816cadd33d79b5610352c5e5c6ae535/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/9a49fa8d68f1800b095c0e1dcb3927d0b2d488fb44e59a0dd31bcdb021afcc71/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/7051c118f8c906d7832c257e0ac0b17a4be0b8cd75e2cc12df3ce3f2465c92d6/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/05bbac6ee220154c57896c3787335fdd8b436394a905639777c6965d6dd70e88/merged 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:41:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:41:47.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:47.295 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-16T09:41:47.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:2024-03-16T09:41:47.297+0000 7ff316e0c480 -1 init, newargv = 0x558ac746d310 newargc=15 2024-03-16T09:41:47.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[84414]: starting ceph client 2024-03-16T09:41:47.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[84414]: starting fuse 2024-03-16T09:41:48.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe fuse 2024-03-16T09:41:48.450 DEBUG:teuthology.orchestra.run.smithi077:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T09:41:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:/proc 2024-03-16T09:41:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:/sys 2024-03-16T09:41:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:/dev 2024-03-16T09:41:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/security 2024-03-16T09:41:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/shm 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/pts 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/run 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/cgroup 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/pstore 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/bpf 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/ 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/selinux 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/hugepages 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/mqueue 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/debug 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/tracing 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/fs/fuse/connections 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T09:41:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:/sys/kernel/config 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/1000 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/ceph 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/906e5dbc089b0a3116389086de52a6621feecce3a9fe9ff27891d19b58cd7a16/merged 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/1ef34430c0ce9d442329192462963be97d3df4fa4c57a0a60cd6a1fad936d13d/merged 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/run/user/0 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/41a6415788e4c303ad62c44f76fdf2f33cf856e7a8b3fb46e0317e8fa5f92bdb/merged 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/236809ad60caee1cddc5b4db6dae2f85c45c13ef6679528058fc24f8dc0a7a21/merged 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/dbe0827f342ef04efe291b0ae1bfb60ea9c97e38cfa24c319d4a54d84f46e572/merged 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/85efabf09ed948bea2ffcc3baeed72e21f2310045b9d1b6a92bd3d170d0bbcd1/merged 2024-03-16T09:41:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/bff085ccab23f994d13295f2d9cc3e5a8816cadd33d79b5610352c5e5c6ae535/merged 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/9a49fa8d68f1800b095c0e1dcb3927d0b2d488fb44e59a0dd31bcdb021afcc71/merged 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/7051c118f8c906d7832c257e0ac0b17a4be0b8cd75e2cc12df3ce3f2465c92d6/merged 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:/var/lib/containers/storage/overlay/05bbac6ee220154c57896c3787335fdd8b436394a905639777c6965d6dd70e88/merged 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:48.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:48.511 DEBUG:teuthology.orchestra.run.smithi077:> ls /sys/fs/fuse/connections 2024-03-16T09:41:48.565 INFO:teuthology.orchestra.run.smithi077.stdout:119 2024-03-16T09:41:48.565 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-16T09:41:48.565 DEBUG:teuthology.orchestra.run.smithi077:> sudo stdin-killer -- python3 -c ' 2024-03-16T09:41:48.565 DEBUG:teuthology.orchestra.run.smithi077:> import glob 2024-03-16T09:41:48.565 DEBUG:teuthology.orchestra.run.smithi077:> import re 2024-03-16T09:41:48.565 DEBUG:teuthology.orchestra.run.smithi077:> import os 2024-03-16T09:41:48.565 DEBUG:teuthology.orchestra.run.smithi077:> import subprocess 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> def _find_admin_socket(client_name): 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> files = glob.glob(asok_path) 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> # Given a non-glob path, it better be there 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> if "*" not in asok_path: 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> assert(len(files) == 1) 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> return files[0] 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> for f in files: 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-16T09:41:48.566 DEBUG:teuthology.orchestra.run.smithi077:> contents = proc_f.read() 2024-03-16T09:41:48.567 DEBUG:teuthology.orchestra.run.smithi077:> if mountpoint in contents: 2024-03-16T09:41:48.567 DEBUG:teuthology.orchestra.run.smithi077:> return f 2024-03-16T09:41:48.567 DEBUG:teuthology.orchestra.run.smithi077:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-16T09:41:48.567 DEBUG:teuthology.orchestra.run.smithi077:> 2024-03-16T09:41:48.567 DEBUG:teuthology.orchestra.run.smithi077:> print(_find_admin_socket("client.0")) 2024-03-16T09:41:48.567 DEBUG:teuthology.orchestra.run.smithi077:> ' 2024-03-16T09:41:48.691 INFO:teuthology.orchestra.run.smithi077.stdout:/var/run/ceph/ceph-client.0.84414.asok 2024-03-16T09:41:48.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T09:41:48.702 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.84414.asok 2024-03-16T09:41:48.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:48.702 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.84414.asok status 2024-03-16T09:41:48.838 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "metadata": { 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "entity_id": "0", 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": "smithi077", 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "pid": "84414", 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "root": "/volumes/qa/sv_0/d6cb7cbe-a826-46f6-811e-8b60ec9c6a86" 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_count": 0, 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "dentry_pinned_count": 0, 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 15255, 2024-03-16T09:41:48.839 INFO:teuthology.orchestra.run.smithi077.stdout: "inst": { 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "name": { 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "client", 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "num": 15255 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 2353223335 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": { 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "192.168.0.1:0", 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 2353223335 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "inst_str": "client.15255 192.168.0.1:0/2353223335", 2024-03-16T09:41:48.840 INFO:teuthology.orchestra.run.smithi077.stdout: "addr_str": "192.168.0.1:0/2353223335", 2024-03-16T09:41:48.841 INFO:teuthology.orchestra.run.smithi077.stdout: "inode_count": 5, 2024-03-16T09:41:48.841 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_epoch": 20, 2024-03-16T09:41:48.841 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch": 88, 2024-03-16T09:41:48.841 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_epoch_barrier": 0, 2024-03-16T09:41:48.841 INFO:teuthology.orchestra.run.smithi077.stdout: "blocklisted": false, 2024-03-16T09:41:48.841 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs" 2024-03-16T09:41:48.841 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:48.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:48.847 DEBUG:teuthology.orchestra.run.smithi077:> stat --file-system '--printf=%T 2024-03-16T09:41:48.847 DEBUG:teuthology.orchestra.run.smithi077:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:48.865 INFO:teuthology.orchestra.run.smithi077.stdout:fuseblk 2024-03-16T09:41:48.865 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:48.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:48.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:48.933 INFO:teuthology.run_tasks:Running task exec... 2024-03-16T09:41:48.944 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T09:41:48.944 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:41:48.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-16T09:41:48.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:48 smithi077 ceph-mon[32171]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:49.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.073+0000 7fcb2c75b640 1 -- 172.21.15.77:0/840899675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb240ffaf0 msgr2=0x7fcb240fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.073+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/840899675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb240ffaf0 0x7fcb240fff10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009cd0 tx=0x7fcb2002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.073+0000 7fcb2c75b640 1 -- 172.21.15.77:0/840899675 shutdown_connections 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.073+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/840899675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb24103480 0x7fcb24109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.073+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/840899675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb24102ac0 0x7fcb24102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.073+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/840899675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb240ffaf0 0x7fcb240fff10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.073+0000 7fcb2c75b640 1 -- 172.21.15.77:0/840899675 >> 172.21.15.77:0/840899675 conn(0x7fcb240fb440 msgr2=0x7fcb240fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 -- 172.21.15.77:0/840899675 shutdown_connections 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 -- 172.21.15.77:0/840899675 wait complete. 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 Processor -- start 2024-03-16T09:41:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 -- start start 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb240ffaf0 0x7fcb24197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb24102ac0 0x7fcb24197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb24103480 0x7fcb2419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2410d370 con 0x7fcb24103480 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2410d1f0 con 0x7fcb240ffaf0 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2c75b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2410d4f0 con 0x7fcb24102ac0 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2acf8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb24103480 0x7fcb2419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2acf8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb24103480 0x7fcb2419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36868/0 (socket says 172.21.15.77:36868) 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2acf8640 1 -- 172.21.15.77:0/1133858264 learned_addr learned my addr 172.21.15.77:0/1133858264 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb2a4f7640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb240ffaf0 0x7fcb24197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.074+0000 7fcb29cf6640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb24102ac0 0x7fcb24197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2acf8640 1 -- 172.21.15.77:0/1133858264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb24102ac0 msgr2=0x7fcb24197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2acf8640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb24102ac0 0x7fcb24197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2acf8640 1 -- 172.21.15.77:0/1133858264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb240ffaf0 msgr2=0x7fcb24197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2acf8640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb240ffaf0 0x7fcb24197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2acf8640 1 -- 172.21.15.77:0/1133858264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2419f630 con 0x7fcb24103480 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2acf8640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb24103480 0x7fcb2419efc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e970 tx=0x7fcb1c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb1b7fe640 1 -- 172.21.15.77:0/1133858264 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c00ccc0 con 0x7fcb24103480 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb1b7fe640 1 -- 172.21.15.77:0/1133858264 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb1c00ce60 con 0x7fcb24103480 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb2419f920 con 0x7fcb24103480 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb1b7fe640 1 -- 172.21.15.77:0/1133858264 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c010430 con 0x7fcb24103480 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.075+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb2419fe70 con 0x7fcb24103480 2024-03-16T09:41:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.076+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb24108c40 con 0x7fcb24103480 2024-03-16T09:41:49.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.077+0000 7fcb1b7fe640 1 -- 172.21.15.77:0/1133858264 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcb1c004fb0 con 0x7fcb24103480 2024-03-16T09:41:49.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.078+0000 7fcb1b7fe640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb100a2920 0x7fcb100a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:49.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.078+0000 7fcb1b7fe640 1 -- 172.21.15.77:0/1133858264 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcb1c0cef90 con 0x7fcb24103480 2024-03-16T09:41:49.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.078+0000 7fcb2a4f7640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb100a2920 0x7fcb100a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:49.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.078+0000 7fcb2a4f7640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb100a2920 0x7fcb100a4de0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcb2002c700 tx=0x7fcb20000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.081+0000 7fcb1b7fe640 1 -- 172.21.15.77:0/1133858264 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb1c053740 con 0x7fcb24103480 2024-03-16T09:41:49.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:48 smithi119 ceph-mon[35632]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:49.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:48 smithi129 ceph-mon[35841]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:41:49.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.294+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fcb2402d400 con 0x7fcb24103480 2024-03-16T09:41:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.860+0000 7fcb1b7fe640 1 -- 172.21.15.77:0/1133858264 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v21) ==== 117+0+0 (secure 0 0 0) 0x7fcb1c09c0e0 con 0x7fcb24103480 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb100a2920 msgr2=0x7fcb100a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb100a2920 0x7fcb100a4de0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcb2002c700 tx=0x7fcb20000f40 comp rx=0 tx=0).stop 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb24103480 msgr2=0x7fcb2419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb24103480 0x7fcb2419efc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e970 tx=0x7fcb1c00ed70 comp rx=0 tx=0).stop 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 shutdown_connections 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcb100a2920 0x7fcb100a4de0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb24103480 0x7fcb2419efc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb24102ac0 0x7fcb24197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 --2- 172.21.15.77:0/1133858264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb240ffaf0 0x7fcb24197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 >> 172.21.15.77:0/1133858264 conn(0x7fcb240fb440 msgr2=0x7fcb24105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 shutdown_connections 2024-03-16T09:41:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:49.863+0000 7fcb2c75b640 1 -- 172.21.15.77:0/1133858264 wait complete. 2024-03-16T09:41:49.877 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-16T09:41:49.888 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-16T09:41:49.898 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 -- 172.21.15.77:0/2299115894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8107560 msgr2=0x7f3ae81099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 --2- 172.21.15.77:0/2299115894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8107560 0x7f3ae81099b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009ca0 tx=0x7f3ae002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 -- 172.21.15.77:0/2299115894 shutdown_connections 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 --2- 172.21.15.77:0/2299115894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8107560 0x7f3ae81099b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 --2- 172.21.15.77:0/2299115894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ae8104c30 0x7f3ae8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 --2- 172.21.15.77:0/2299115894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ae8102300 0x7f3ae81046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 -- 172.21.15.77:0/2299115894 >> 172.21.15.77:0/2299115894 conn(0x7f3ae80fbd10 msgr2=0x7f3ae80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:50.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 -- 172.21.15.77:0/2299115894 shutdown_connections 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.019+0000 7f3af0352640 1 -- 172.21.15.77:0/2299115894 wait complete. 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 Processor -- start 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 -- start start 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8102300 0x7f3ae8101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ae8104c30 0x7f3ae81002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ae8107560 0x7f3ae81007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae810d360 con 0x7f3ae8102300 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae810d1e0 con 0x7f3ae8107560 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3af0352640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae810d4e0 con 0x7f3ae8104c30 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8102300 0x7f3ae8101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8102300 0x7f3ae8101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36896/0 (socket says 172.21.15.77:36896) 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 -- 172.21.15.77:0/3350836014 learned_addr learned my addr 172.21.15.77:0/3350836014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aed8ed640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ae8104c30 0x7f3ae81002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee8ef640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ae8107560 0x7f3ae81007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 -- 172.21.15.77:0/3350836014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ae8104c30 msgr2=0x7f3ae81002b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ae8104c30 0x7f3ae81002b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 -- 172.21.15.77:0/3350836014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ae8107560 msgr2=0x7f3ae81007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ae8107560 0x7f3ae81007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.020+0000 7f3aee0ee640 1 -- 172.21.15.77:0/3350836014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae8101080 con 0x7f3ae8102300 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.021+0000 7f3aee0ee640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8102300 0x7f3ae8101c70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400e9f0 tx=0x7f3ae400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.021+0000 7f3adf7fe640 1 -- 172.21.15.77:0/3350836014 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae400cd80 con 0x7f3ae8102300 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.021+0000 7f3adf7fe640 1 -- 172.21.15.77:0/3350836014 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ae40108e0 con 0x7f3ae8102300 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.021+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae81a1b00 con 0x7f3ae8102300 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.021+0000 7f3adf7fe640 1 -- 172.21.15.77:0/3350836014 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4018920 con 0x7f3ae8102300 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.021+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae81a2050 con 0x7f3ae8102300 2024-03-16T09:41:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.022+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae80fd800 con 0x7f3ae8102300 2024-03-16T09:41:50.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.023+0000 7f3adf7fe640 1 -- 172.21.15.77:0/3350836014 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3ae4010a90 con 0x7f3ae8102300 2024-03-16T09:41:50.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.024+0000 7f3adf7fe640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ad40a2800 0x7f3ad40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.024+0000 7f3aed8ed640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ad40a2800 0x7f3ad40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.024+0000 7f3adf7fe640 1 -- 172.21.15.77:0/3350836014 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3ae40cf210 con 0x7f3ae8102300 2024-03-16T09:41:50.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.024+0000 7f3aed8ed640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ad40a2800 0x7f3ad40a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ae81019e0 tx=0x7f3ad8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.027+0000 7f3adf7fe640 1 -- 172.21.15.77:0/3350836014 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ae409bc70 con 0x7f3ae8102300 2024-03-16T09:41:50.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:49 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1133858264' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T09:41:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.202+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ae802d380 con 0x7f3ae8102300 2024-03-16T09:41:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.204+0000 7f3adf7fe640 1 -- 172.21.15.77:0/3350836014 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f3ae802d380 con 0x7f3ae8102300 2024-03-16T09:41:50.207 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:50.207 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ad40a2800 msgr2=0x7f3ad40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ad40a2800 0x7f3ad40a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ae81019e0 tx=0x7f3ad8005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8102300 msgr2=0x7f3ae8101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8102300 0x7f3ae8101c70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400e9f0 tx=0x7f3ae400eeb0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 shutdown_connections 2024-03-16T09:41:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3ad40a2800 0x7f3ad40a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ae8107560 0x7f3ae81007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ae8104c30 0x7f3ae81002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 --2- 172.21.15.77:0/3350836014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ae8102300 0x7f3ae8101c70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 >> 172.21.15.77:0/3350836014 conn(0x7f3ae80fbd10 msgr2=0x7f3ae8102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:50.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 shutdown_connections 2024-03-16T09:41:50.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.206+0000 7f3af0352640 1 -- 172.21.15.77:0/3350836014 wait complete. 2024-03-16T09:41:50.219 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:49 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1133858264' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T09:41:50.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:41:50.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:41:50.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:49 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1133858264' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-16T09:41:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 -- 172.21.15.77:0/4201775940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb414102290 msgr2=0x7fb414108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4201775940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb414102290 0x7fb414108b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009860 tx=0x7fb40c02bb70 comp rx=0 tx=0).stop 2024-03-16T09:41:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 -- 172.21.15.77:0/4201775940 shutdown_connections 2024-03-16T09:41:50.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4201775940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb414102290 0x7fb414108b20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4201775940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141018d0 0x7fb414101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4201775940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4141006e0 0x7fb414100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 -- 172.21.15.77:0/4201775940 >> 172.21.15.77:0/4201775940 conn(0x7fb4140fbc70 msgr2=0x7fb4140fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 -- 172.21.15.77:0/4201775940 shutdown_connections 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.321+0000 7fb4198ac640 1 -- 172.21.15.77:0/4201775940 wait complete. 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 Processor -- start 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 -- start start 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141006e0 0x7fb4141979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4141018d0 0x7fb414197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb414102290 0x7fb41419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb41410d330 con 0x7fb4141006e0 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb41410d1b0 con 0x7fb414102290 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4198ac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb41410d4b0 con 0x7fb4141018d0 2024-03-16T09:41:50.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb412ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141006e0 0x7fb4141979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb412ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141006e0 0x7fb4141979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36916/0 (socket says 172.21.15.77:36916) 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb412ffd640 1 -- 172.21.15.77:0/4059165059 learned_addr learned my addr 172.21.15.77:0/4059165059 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4127fc640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4141018d0 0x7fb414197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.322+0000 7fb4137fe640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb414102290 0x7fb41419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb412ffd640 1 -- 172.21.15.77:0/4059165059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4141018d0 msgr2=0x7fb414197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb412ffd640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4141018d0 0x7fb414197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb412ffd640 1 -- 172.21.15.77:0/4059165059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb414102290 msgr2=0x7fb41419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb412ffd640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb414102290 0x7fb41419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb412ffd640 1 -- 172.21.15.77:0/4059165059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41419f610 con 0x7fb4141006e0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb412ffd640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141006e0 0x7fb4141979f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb40400c3f0 tx=0x7fb404009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb4188aa640 1 -- 172.21.15.77:0/4059165059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40401b070 con 0x7fb4141006e0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb4188aa640 1 -- 172.21.15.77:0/4059165059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb40400bae0 con 0x7fb4141006e0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41419f900 con 0x7fb4141006e0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb4188aa640 1 -- 172.21.15.77:0/4059165059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb404016c50 con 0x7fb4141006e0 2024-03-16T09:41:50.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.323+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41419fe50 con 0x7fb4141006e0 2024-03-16T09:41:50.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.324+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb414107a50 con 0x7fb4141006e0 2024-03-16T09:41:50.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.325+0000 7fb4188aa640 1 -- 172.21.15.77:0/4059165059 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb40401e070 con 0x7fb4141006e0 2024-03-16T09:41:50.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.326+0000 7fb4188aa640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3fc0a2800 0x7fb3fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.326+0000 7fb4127fc640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3fc0a2800 0x7fb3fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.326+0000 7fb4188aa640 1 -- 172.21.15.77:0/4059165059 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb4040cef90 con 0x7fb4141006e0 2024-03-16T09:41:50.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.326+0000 7fb4127fc640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3fc0a2800 0x7fb3fc0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb414198f00 tx=0x7fb408005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.329+0000 7fb4188aa640 1 -- 172.21.15.77:0/4059165059 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb404025080 con 0x7fb4141006e0 2024-03-16T09:41:50.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.503+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb41402d410 con 0x7fb4141006e0 2024-03-16T09:41:50.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.504+0000 7fb4188aa640 1 -- 172.21.15.77:0/4059165059 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fb40409b9f0 con 0x7fb4141006e0 2024-03-16T09:41:50.507 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3fc0a2800 msgr2=0x7fb3fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3fc0a2800 0x7fb3fc0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb414198f00 tx=0x7fb408005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141006e0 msgr2=0x7fb4141979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141006e0 0x7fb4141979f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb40400c3f0 tx=0x7fb404009f90 comp rx=0 tx=0).stop 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 shutdown_connections 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb3fc0a2800 0x7fb3fc0a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb414102290 0x7fb41419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4141018d0 0x7fb414197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 --2- 172.21.15.77:0/4059165059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4141006e0 0x7fb4141979f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 >> 172.21.15.77:0/4059165059 conn(0x7fb4140fbc70 msgr2=0x7fb414104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 shutdown_connections 2024-03-16T09:41:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.506+0000 7fb4198ac640 1 -- 172.21.15.77:0/4059165059 wait complete. 2024-03-16T09:41:50.519 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:41:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 -- 172.21.15.77:0/2951204276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 msgr2=0x7f408c108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 --2- 172.21.15.77:0/2951204276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 0x7f408c108c20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4084009ca0 tx=0x7f408402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 -- 172.21.15.77:0/2951204276 shutdown_connections 2024-03-16T09:41:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 --2- 172.21.15.77:0/2951204276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 0x7f408c108c20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 --2- 172.21.15.77:0/2951204276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 0x7f408c100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 --2- 172.21.15.77:0/2951204276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c101c90 0x7f408c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 -- 172.21.15.77:0/2951204276 >> 172.21.15.77:0/2951204276 conn(0x7f408c0fb410 msgr2=0x7f408c0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 -- 172.21.15.77:0/2951204276 shutdown_connections 2024-03-16T09:41:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.621+0000 7f409426e640 1 -- 172.21.15.77:0/2951204276 wait complete. 2024-03-16T09:41:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 Processor -- start 2024-03-16T09:41:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 -- start start 2024-03-16T09:41:50.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 0x7f408c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 0x7f408c196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c101c90 0x7f408c19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f408c10c230 con 0x7f408c101380 2024-03-16T09:41:50.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f408c10c0b0 con 0x7f408c1009c0 2024-03-16T09:41:50.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409426e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f408c10c3b0 con 0x7f408c101c90 2024-03-16T09:41:50.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f4091809640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 0x7f408c196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 0x7f408c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 0x7f408c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:51878/0 (socket says 172.21.15.77:51878) 2024-03-16T09:41:50.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 -- 172.21.15.77:0/3827366766 learned_addr learned my addr 172.21.15.77:0/3827366766 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:50.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409280b640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c101c90 0x7f408c19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 -- 172.21.15.77:0/3827366766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c101c90 msgr2=0x7f408c19de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c101c90 0x7f408c19de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 -- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 msgr2=0x7f408c196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 0x7f408c196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 -- 172.21.15.77:0/3827366766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c19e4f0 con 0x7f408c1009c0 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f4091809640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 0x7f408c196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f409200a640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 0x7f408c1968d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f408800e9c0 tx=0x7f408800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.622+0000 7f40837fe640 1 -- 172.21.15.77:0/3827366766 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408800cd80 con 0x7f408c1009c0 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.623+0000 7f40837fe640 1 -- 172.21.15.77:0/3827366766 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40880035f0 con 0x7f408c1009c0 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.623+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c19e7e0 con 0x7f408c1009c0 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.623+0000 7f40837fe640 1 -- 172.21.15.77:0/3827366766 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088010690 con 0x7f408c1009c0 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.623+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c19ece0 con 0x7f408c1009c0 2024-03-16T09:41:50.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.623+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f408c107b80 con 0x7f408c1009c0 2024-03-16T09:41:50.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.625+0000 7f40837fe640 1 -- 172.21.15.77:0/3827366766 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f40880037a0 con 0x7f408c1009c0 2024-03-16T09:41:50.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.625+0000 7f40837fe640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40780a2800 0x7f40780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.625+0000 7f4091809640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40780a2800 0x7f40780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.625+0000 7f40837fe640 1 -- 172.21.15.77:0/3827366766 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f40880cf120 con 0x7f408c1009c0 2024-03-16T09:41:50.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.626+0000 7f4091809640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40780a2800 0x7f40780a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f407c0095f0 tx=0x7f407c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.630+0000 7f40837fe640 1 -- 172.21.15.77:0/3827366766 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f408809c170 con 0x7f408c1009c0 2024-03-16T09:41:50.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.806+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f408c0623e0 con 0x7f408c1009c0 2024-03-16T09:41:50.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.806+0000 7f40837fe640 1 -- 172.21.15.77:0/3827366766 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f408809ba70 con 0x7f408c1009c0 2024-03-16T09:41:50.809 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:41:50.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40780a2800 msgr2=0x7f40780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40780a2800 0x7f40780a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f407c0095f0 tx=0x7f407c005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:50.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 msgr2=0x7f408c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 0x7f408c1968d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f408800e9c0 tx=0x7f408800ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 shutdown_connections 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40780a2800 0x7f40780a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c101c90 0x7f408c19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f408c101380 0x7f408c196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 --2- 172.21.15.77:0/3827366766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f408c1009c0 0x7f408c1968d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 >> 172.21.15.77:0/3827366766 conn(0x7f408c0fb410 msgr2=0x7f408c104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 shutdown_connections 2024-03-16T09:41:50.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.808+0000 7f409426e640 1 -- 172.21.15.77:0/3827366766 wait complete. 2024-03-16T09:41:50.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.924+0000 7f2785753640 1 -- 172.21.15.77:0/3673819454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801018f0 msgr2=0x7f2780101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.924+0000 7f2785753640 1 --2- 172.21.15.77:0/3673819454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801018f0 0x7f2780101d70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2774009ca0 tx=0x7f277402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.924+0000 7f2785753640 1 -- 172.21.15.77:0/3673819454 shutdown_connections 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.924+0000 7f2785753640 1 --2- 172.21.15.77:0/3673819454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f27801022b0 0x7f2780108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.924+0000 7f2785753640 1 --2- 172.21.15.77:0/3673819454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801018f0 0x7f2780101d70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.924+0000 7f2785753640 1 --2- 172.21.15.77:0/3673819454 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2780100700 0x7f2780100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 -- 172.21.15.77:0/3673819454 >> 172.21.15.77:0/3673819454 conn(0x7f27800fbc70 msgr2=0x7f27800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 -- 172.21.15.77:0/3673819454 shutdown_connections 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 -- 172.21.15.77:0/3673819454 wait complete. 2024-03-16T09:41:50.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 Processor -- start 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 -- start start 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2780100700 0x7f2780197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f27801018f0 0x7f2780198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801022b0 0x7f278019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f278010c230 con 0x7f27801022b0 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f278010c0b0 con 0x7f2780100700 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f2785753640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f278010c3b0 con 0x7f27801018f0 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.925+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801022b0 0x7f278019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277effd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2780100700 0x7f2780197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801022b0 0x7f278019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36950/0 (socket says 172.21.15.77:36950) 2024-03-16T09:41:50.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 -- 172.21.15.77:0/2440327192 learned_addr learned my addr 172.21.15.77:0/2440327192 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 -- 172.21.15.77:0/2440327192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f27801018f0 msgr2=0x7f2780198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f27801018f0 0x7f2780198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 -- 172.21.15.77:0/2440327192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2780100700 msgr2=0x7f2780197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2780100700 0x7f2780197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 -- 172.21.15.77:0/2440327192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278019f710 con 0x7f27801022b0 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f277f7fe640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801022b0 0x7f278019f0a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f277000e970 tx=0x7f277000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f2767fff640 1 -- 172.21.15.77:0/2440327192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277000ccc0 con 0x7f27801022b0 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f2767fff640 1 -- 172.21.15.77:0/2440327192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f277000ce60 con 0x7f27801022b0 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f2767fff640 1 -- 172.21.15.77:0/2440327192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770010430 con 0x7f27801022b0 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278019fa00 con 0x7f27801022b0 2024-03-16T09:41:50.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.926+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278019ff00 con 0x7f27801022b0 2024-03-16T09:41:50.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.927+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27801a0190 con 0x7f27801022b0 2024-03-16T09:41:50.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.928+0000 7f2767fff640 1 -- 172.21.15.77:0/2440327192 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2770004f90 con 0x7f27801022b0 2024-03-16T09:41:50.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.929+0000 7f2767fff640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f27680a2800 0x7f27680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:50.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.929+0000 7f277effd640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f27680a2800 0x7f27680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:50.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.929+0000 7f2767fff640 1 -- 172.21.15.77:0/2440327192 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f27700ceea0 con 0x7f27801022b0 2024-03-16T09:41:50.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.929+0000 7f277effd640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f27680a2800 0x7f27680a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2780101750 tx=0x7f277800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:50.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:50.933+0000 7f2767fff640 1 -- 172.21.15.77:0/2440327192 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f277009bf50 con 0x7f27801022b0 2024-03-16T09:41:51.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.112+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f27801a0360 con 0x7f27801022b0 2024-03-16T09:41:51.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.112+0000 7f2767fff640 1 -- 172.21.15.77:0/2440327192 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f277009b850 con 0x7f27801022b0 2024-03-16T09:41:51.115 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:41:51.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f27680a2800 msgr2=0x7f27680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f27680a2800 0x7f27680a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2780101750 tx=0x7f277800b390 comp rx=0 tx=0).stop 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801022b0 msgr2=0x7f278019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801022b0 0x7f278019f0a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f277000e970 tx=0x7f277000ed70 comp rx=0 tx=0).stop 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 shutdown_connections 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f27680a2800 0x7f27680a4cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27801022b0 0x7f278019f0a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f27801018f0 0x7f2780198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 --2- 172.21.15.77:0/2440327192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2780100700 0x7f2780197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 >> 172.21.15.77:0/2440327192 conn(0x7f27800fbc70 msgr2=0x7f2780069b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 shutdown_connections 2024-03-16T09:41:51.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.114+0000 7f2785753640 1 -- 172.21.15.77:0/2440327192 wait complete. 2024-03-16T09:41:51.127 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:50 smithi077 ceph-mon[32171]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:41:51.127 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:50 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1133858264' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-16T09:41:51.128 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:50 smithi077 ceph-mon[32171]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T09:41:51.128 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:50 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3350836014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:51.128 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:50 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4059165059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:51.128 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:50 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3827366766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:51.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:51.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:50 smithi119 ceph-mon[35632]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:41:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:50 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1133858264' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-16T09:41:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:50 smithi119 ceph-mon[35632]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T09:41:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:50 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3350836014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:50 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4059165059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:50 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3827366766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 -- 172.21.15.77:0/338502478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a00ff860 msgr2=0x7f36a00ffce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 --2- 172.21.15.77:0/338502478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a00ff860 0x7f36a00ffce0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3694009900 tx=0x7f369402bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 -- 172.21.15.77:0/338502478 shutdown_connections 2024-03-16T09:41:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 --2- 172.21.15.77:0/338502478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36a0100220 0x7f36a0109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 --2- 172.21.15.77:0/338502478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a00ff860 0x7f36a00ffce0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 --2- 172.21.15.77:0/338502478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36a0102df0 0x7f36a0103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 -- 172.21.15.77:0/338502478 >> 172.21.15.77:0/338502478 conn(0x7f36a00fb400 msgr2=0x7f36a00fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 -- 172.21.15.77:0/338502478 shutdown_connections 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 -- 172.21.15.77:0/338502478 wait complete. 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 Processor -- start 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.227+0000 7f36a8636640 1 -- start start 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36a00ff860 0x7f36a01024f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36a0100220 0x7f36a0100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a0102df0 0x7f36a0101070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36a010d350 con 0x7f36a0102df0 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f36a010d1d0 con 0x7f36a0100220 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36a010d4d0 con 0x7f36a00ff860 2024-03-16T09:41:51.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a0102df0 0x7f36a0101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a0102df0 0x7f36a0101070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36972/0 (socket says 172.21.15.77:36972) 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 -- 172.21.15.77:0/3621067997 learned_addr learned my addr 172.21.15.77:0/3621067997 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a63d2640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36a00ff860 0x7f36a01024f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a5bd1640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36a0100220 0x7f36a0100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 -- 172.21.15.77:0/3621067997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36a00ff860 msgr2=0x7f36a01024f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36a00ff860 0x7f36a01024f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 -- 172.21.15.77:0/3621067997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36a0100220 msgr2=0x7f36a0100b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36a0100220 0x7f36a0100b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 -- 172.21.15.77:0/3621067997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a0101900 con 0x7f36a0102df0 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a5bd1640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36a0100220 0x7f36a0100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a6bd3640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a0102df0 0x7f36a0101070 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f369000e970 tx=0x7f369000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f369f7fe640 1 -- 172.21.15.77:0/3621067997 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369000ccc0 con 0x7f36a0102df0 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f369f7fe640 1 -- 172.21.15.77:0/3621067997 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f369000ce60 con 0x7f36a0102df0 2024-03-16T09:41:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a01a1b40 con 0x7f36a0102df0 2024-03-16T09:41:51.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f369f7fe640 1 -- 172.21.15.77:0/3621067997 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690010430 con 0x7f36a0102df0 2024-03-16T09:41:51.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.228+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a01a2040 con 0x7f36a0102df0 2024-03-16T09:41:51.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.229+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a0108d80 con 0x7f36a0102df0 2024-03-16T09:41:51.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.230+0000 7f369f7fe640 1 -- 172.21.15.77:0/3621067997 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3690004ff0 con 0x7f36a0102df0 2024-03-16T09:41:51.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.231+0000 7f369f7fe640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f36880a2800 0x7f36880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.231+0000 7f36a63d2640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f36880a2800 0x7f36880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.231+0000 7f369f7fe640 1 -- 172.21.15.77:0/3621067997 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f36900cef40 con 0x7f36a0102df0 2024-03-16T09:41:51.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.231+0000 7f36a63d2640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f36880a2800 0x7f36880a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f36a0069bb0 tx=0x7f3698009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:51.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.235+0000 7f369f7fe640 1 -- 172.21.15.77:0/3621067997 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f369009bff0 con 0x7f36a0102df0 2024-03-16T09:41:51.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:50 smithi129 ceph-mon[35841]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T09:41:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:50 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1133858264' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-16T09:41:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:50 smithi129 ceph-mon[35841]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T09:41:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:50 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3350836014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:50 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4059165059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:51.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:50 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3827366766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:51.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.413+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36a0062480 con 0x7f36a0102df0 2024-03-16T09:41:51.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.414+0000 7f369f7fe640 1 -- 172.21.15.77:0/3621067997 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f369009b8f0 con 0x7f36a0102df0 2024-03-16T09:41:51.417 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:51.418 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:51.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f36880a2800 msgr2=0x7f36880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f36880a2800 0x7f36880a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f36a0069bb0 tx=0x7f3698009500 comp rx=0 tx=0).stop 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a0102df0 msgr2=0x7f36a0101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a0102df0 0x7f36a0101070 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f369000e970 tx=0x7f369000ed70 comp rx=0 tx=0).stop 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 shutdown_connections 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f36880a2800 0x7f36880a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a0102df0 0x7f36a0101070 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36a0100220 0x7f36a0100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 --2- 172.21.15.77:0/3621067997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36a00ff860 0x7f36a01024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.416+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 >> 172.21.15.77:0/3621067997 conn(0x7f36a00fb400 msgr2=0x7f36a0105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.417+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 shutdown_connections 2024-03-16T09:41:51.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.417+0000 7f36a8636640 1 -- 172.21.15.77:0/3621067997 wait complete. 2024-03-16T09:41:51.431 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:41:51.431 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:41:51.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 -- 172.21.15.77:0/526347984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 msgr2=0x7fabec101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/526347984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec101d70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009900 tx=0x7fabdc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 -- 172.21.15.77:0/526347984 shutdown_connections 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/526347984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabec1022b0 0x7fabec108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/526347984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec101d70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/526347984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabec100700 0x7fabec100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 -- 172.21.15.77:0/526347984 >> 172.21.15.77:0/526347984 conn(0x7fabec0fbc90 msgr2=0x7fabec0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 -- 172.21.15.77:0/526347984 shutdown_connections 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.532+0000 7fabf0ff4640 1 -- 172.21.15.77:0/526347984 wait complete. 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 Processor -- start 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 -- start start 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabec100700 0x7fabec197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabec1022b0 0x7fabec19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fabec10c230 con 0x7fabec1018f0 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fabec10c0b0 con 0x7fabec1022b0 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabf0ff4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fabec10c3b0 con 0x7fabec100700 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36980/0 (socket says 172.21.15.77:36980) 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 -- 172.21.15.77:0/604346722 learned_addr learned my addr 172.21.15.77:0/604346722 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabea59c640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabec100700 0x7fabec197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabead9d640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabec1022b0 0x7fabec19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 -- 172.21.15.77:0/604346722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabec100700 msgr2=0x7fabec197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabec100700 0x7fabec197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 -- 172.21.15.77:0/604346722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabec1022b0 msgr2=0x7fabec19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabec1022b0 0x7fabec19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.533+0000 7fabe9d9b640 1 -- 172.21.15.77:0/604346722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec19f6c0 con 0x7fabec1018f0 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.534+0000 7fabe9d9b640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec198030 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006010 tx=0x7fabdc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:51.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.534+0000 7fabdb7fe640 1 -- 172.21.15.77:0/604346722 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc043020 con 0x7fabec1018f0 2024-03-16T09:41:51.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.534+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec19f950 con 0x7fabec1018f0 2024-03-16T09:41:51.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.534+0000 7fabdb7fe640 1 -- 172.21.15.77:0/604346722 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabdc0043d0 con 0x7fabec1018f0 2024-03-16T09:41:51.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.534+0000 7fabdb7fe640 1 -- 172.21.15.77:0/604346722 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc003a80 con 0x7fabec1018f0 2024-03-16T09:41:51.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.534+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec19fea0 con 0x7fabec1018f0 2024-03-16T09:41:51.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.535+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec068a20 con 0x7fabec1018f0 2024-03-16T09:41:51.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.536+0000 7fabdb7fe640 1 -- 172.21.15.77:0/604346722 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fabdc046020 con 0x7fabec1018f0 2024-03-16T09:41:51.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.536+0000 7fabdb7fe640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabd40a2800 0x7fabd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.536+0000 7fabea59c640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabd40a2800 0x7fabd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.536+0000 7fabdb7fe640 1 -- 172.21.15.77:0/604346722 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fabdc0f0c10 con 0x7fabec1018f0 2024-03-16T09:41:51.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.537+0000 7fabea59c640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabd40a2800 0x7fabd40a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fabec101750 tx=0x7fabe0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:51.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.541+0000 7fabdb7fe640 1 -- 172.21.15.77:0/604346722 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fabdc04d020 con 0x7fabec1018f0 2024-03-16T09:41:51.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.714+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fabec1a0800 con 0x7fabec1018f0 2024-03-16T09:41:51.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.714+0000 7fabdb7fe640 1 -- 172.21.15.77:0/604346722 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fabdc0bdcc0 con 0x7fabec1018f0 2024-03-16T09:41:51.717 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:41:51.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabd40a2800 msgr2=0x7fabd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabd40a2800 0x7fabd40a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fabec101750 tx=0x7fabe0005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:51.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 msgr2=0x7fabec198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec198030 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006010 tx=0x7fabdc005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:51.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 shutdown_connections 2024-03-16T09:41:51.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fabd40a2800 0x7fabd40a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabec1022b0 0x7fabec19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabec1018f0 0x7fabec198030 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.716+0000 7fabf0ff4640 1 --2- 172.21.15.77:0/604346722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabec100700 0x7fabec197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.717+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 >> 172.21.15.77:0/604346722 conn(0x7fabec0fbc90 msgr2=0x7fabec069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:51.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.717+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 shutdown_connections 2024-03-16T09:41:51.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.717+0000 7fabf0ff4640 1 -- 172.21.15.77:0/604346722 wait complete. 2024-03-16T09:41:51.731 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:41:51.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 -- 172.21.15.77:0/2640251186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af81018f0 msgr2=0x7f4af8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 --2- 172.21.15.77:0/2640251186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af81018f0 0x7f4af8101d70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009900 tx=0x7f4ae802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:51.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 -- 172.21.15.77:0/2640251186 shutdown_connections 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 --2- 172.21.15.77:0/2640251186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4af81022b0 0x7f4af8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 --2- 172.21.15.77:0/2640251186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af81018f0 0x7f4af8101d70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 --2- 172.21.15.77:0/2640251186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af8100700 0x7f4af8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 -- 172.21.15.77:0/2640251186 >> 172.21.15.77:0/2640251186 conn(0x7f4af80fbc90 msgr2=0x7f4af80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 -- 172.21.15.77:0/2640251186 shutdown_connections 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 -- 172.21.15.77:0/2640251186 wait complete. 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 Processor -- start 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.833+0000 7f4affab8640 1 -- start start 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af8100700 0x7f4af806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4af81018f0 0x7f4af819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af81022b0 0x7f4af81a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4af810c210 con 0x7f4af8100700 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4af810c090 con 0x7f4af81018f0 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4af810c390 con 0x7f4af81022b0 2024-03-16T09:41:51.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af8100700 0x7f4af806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af8100700 0x7f4af806a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37006/0 (socket says 172.21.15.77:37006) 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 -- 172.21.15.77:0/3354665720 learned_addr learned my addr 172.21.15.77:0/3354665720 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd053640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4af81018f0 0x7f4af819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afe055640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af81022b0 0x7f4af81a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 -- 172.21.15.77:0/3354665720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af81022b0 msgr2=0x7f4af81a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af81022b0 0x7f4af81a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 -- 172.21.15.77:0/3354665720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4af81018f0 msgr2=0x7f4af819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4af81018f0 0x7f4af819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 -- 172.21.15.77:0/3354665720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af81a18f0 con 0x7f4af8100700 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4afd854640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af8100700 0x7f4af806a900 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4af000e9c0 tx=0x7f4af000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4aeeffd640 1 -- 172.21.15.77:0/3354665720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af000cd80 con 0x7f4af8100700 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4aeeffd640 1 -- 172.21.15.77:0/3354665720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4af00035f0 con 0x7f4af8100700 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af81a1b80 con 0x7f4af8100700 2024-03-16T09:41:51.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4aeeffd640 1 -- 172.21.15.77:0/3354665720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0010690 con 0x7f4af8100700 2024-03-16T09:41:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.834+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af81a20a0 con 0x7f4af8100700 2024-03-16T09:41:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.835+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af8107b40 con 0x7f4af8100700 2024-03-16T09:41:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.836+0000 7f4aeeffd640 1 -- 172.21.15.77:0/3354665720 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4af00037a0 con 0x7f4af8100700 2024-03-16T09:41:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.837+0000 7f4aeeffd640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4ae40a2830 0x7f4ae40a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.837+0000 7f4afd053640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4ae40a2830 0x7f4ae40a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:51.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.837+0000 7f4aeeffd640 1 -- 172.21.15.77:0/3354665720 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f4af00ced70 con 0x7f4af8100700 2024-03-16T09:41:51.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.838+0000 7f4afd053640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4ae40a2830 0x7f4ae40a4cf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009230 tx=0x7f4ae8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:51.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:51.842+0000 7f4aeeffd640 1 -- 172.21.15.77:0/3354665720 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4af009be20 con 0x7f4af8100700 2024-03-16T09:41:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:51 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2440327192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:51 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3621067997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:52.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:51 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/604346722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.018+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4af8062480 con 0x7f4af8100700 2024-03-16T09:41:52.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.018+0000 7f4aeeffd640 1 -- 172.21.15.77:0/3354665720 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f4af009b720 con 0x7f4af8100700 2024-03-16T09:41:52.021 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:41:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4ae40a2830 msgr2=0x7f4ae40a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4ae40a2830 0x7f4ae40a4cf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009230 tx=0x7f4ae8005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af8100700 msgr2=0x7f4af806a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af8100700 0x7f4af806a900 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4af000e9c0 tx=0x7f4af000ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 shutdown_connections 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4ae40a2830 0x7f4ae40a4cf0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4af81022b0 0x7f4af81a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4af81018f0 0x7f4af819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 --2- 172.21.15.77:0/3354665720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4af8100700 0x7f4af806a900 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 >> 172.21.15.77:0/3354665720 conn(0x7f4af80fbc90 msgr2=0x7f4af8104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 shutdown_connections 2024-03-16T09:41:52.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.020+0000 7f4affab8640 1 -- 172.21.15.77:0/3354665720 wait complete. 2024-03-16T09:41:52.034 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:41:52.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 -- 172.21.15.77:0/3845390036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 msgr2=0x7facb0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 --2- 172.21.15.77:0/3845390036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0100b20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7faca8009ca0 tx=0x7faca802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 -- 172.21.15.77:0/3845390036 shutdown_connections 2024-03-16T09:41:52.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 --2- 172.21.15.77:0/3845390036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facb01022b0 0x7facb0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 --2- 172.21.15.77:0/3845390036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7facb01018f0 0x7facb0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 --2- 172.21.15.77:0/3845390036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0100b20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 -- 172.21.15.77:0/3845390036 >> 172.21.15.77:0/3845390036 conn(0x7facb00fbc90 msgr2=0x7facb00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 -- 172.21.15.77:0/3845390036 shutdown_connections 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.136+0000 7facb49a9640 1 -- 172.21.15.77:0/3845390036 wait complete. 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 Processor -- start 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 -- start start 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facb01018f0 0x7facb0196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7facb01022b0 0x7facb019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7facb010c200 con 0x7facb0100700 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7facb010c080 con 0x7facb01022b0 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facb49a9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7facb010c380 con 0x7facb01018f0 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37016/0 (socket says 172.21.15.77:37016) 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 -- 172.21.15.77:0/1682060792 learned_addr learned my addr 172.21.15.77:0/1682060792 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facadd9b640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facb01018f0 0x7facb0196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facaed9d640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7facb01022b0 0x7facb019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 -- 172.21.15.77:0/1682060792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facb01018f0 msgr2=0x7facb0196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facb01018f0 0x7facb0196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 -- 172.21.15.77:0/1682060792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7facb01022b0 msgr2=0x7facb019dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7facb01022b0 0x7facb019dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.137+0000 7facae59c640 1 -- 172.21.15.77:0/1682060792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb019e550 con 0x7facb0100700 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.138+0000 7facae59c640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0196930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faca8032380 tx=0x7faca8032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.138+0000 7fac9f7fe640 1 -- 172.21.15.77:0/1682060792 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8043020 con 0x7facb0100700 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.138+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb0069df0 con 0x7facb0100700 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.138+0000 7fac9f7fe640 1 -- 172.21.15.77:0/1682060792 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faca8003e20 con 0x7facb0100700 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.138+0000 7fac9f7fe640 1 -- 172.21.15.77:0/1682060792 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8039040 con 0x7facb0100700 2024-03-16T09:41:52.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.138+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb006a330 con 0x7facb0100700 2024-03-16T09:41:52.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.139+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb0068a20 con 0x7facb0100700 2024-03-16T09:41:52.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.140+0000 7fac9f7fe640 1 -- 172.21.15.77:0/1682060792 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faca8046020 con 0x7facb0100700 2024-03-16T09:41:52.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.140+0000 7fac9f7fe640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac980a2800 0x7fac980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.141+0000 7facadd9b640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac980a2800 0x7fac980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.141+0000 7fac9f7fe640 1 -- 172.21.15.77:0/1682060792 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7faca80f2f30 con 0x7facb0100700 2024-03-16T09:41:52.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.141+0000 7facadd9b640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac980a2800 0x7fac980a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faca40095f0 tx=0x7faca4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:52.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.145+0000 7fac9f7fe640 1 -- 172.21.15.77:0/1682060792 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faca804d020 con 0x7facb0100700 2024-03-16T09:41:52.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:51 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2440327192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:51 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3621067997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:52.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:51 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/604346722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:51 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2440327192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:51 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3621067997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:52.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:51 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/604346722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.319+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7facb019e7e0 con 0x7facb0100700 2024-03-16T09:41:52.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.320+0000 7fac9f7fe640 1 -- 172.21.15.77:0/1682060792 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7faca80bffe0 con 0x7facb0100700 2024-03-16T09:41:52.323 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac980a2800 msgr2=0x7fac980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac980a2800 0x7fac980a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faca40095f0 tx=0x7faca4005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 msgr2=0x7facb0196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0196930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faca8032380 tx=0x7faca8032820 comp rx=0 tx=0).stop 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 shutdown_connections 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac980a2800 0x7fac980a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7facb01022b0 0x7facb019dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facb01018f0 0x7facb0196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 --2- 172.21.15.77:0/1682060792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100700 0x7facb0196930 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 >> 172.21.15.77:0/1682060792 conn(0x7facb00fbc90 msgr2=0x7facb0104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 shutdown_connections 2024-03-16T09:41:52.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.322+0000 7facb49a9640 1 -- 172.21.15.77:0/1682060792 wait complete. 2024-03-16T09:41:52.335 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-16T09:41:52.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 -- 172.21.15.77:0/234969903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 msgr2=0x7f4048100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 --2- 172.21.15.77:0/234969903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048100b20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4038009900 tx=0x7f403802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 -- 172.21.15.77:0/234969903 shutdown_connections 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 --2- 172.21.15.77:0/234969903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40481022b0 0x7f4048108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 --2- 172.21.15.77:0/234969903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40481018f0 0x7f4048101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 --2- 172.21.15.77:0/234969903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048100b20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 -- 172.21.15.77:0/234969903 >> 172.21.15.77:0/234969903 conn(0x7f40480fbc70 msgr2=0x7f40480fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 -- 172.21.15.77:0/234969903 shutdown_connections 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.433+0000 7f404e113640 1 -- 172.21.15.77:0/234969903 wait complete. 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 Processor -- start 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 -- start start 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40481018f0 0x7f4048197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40481022b0 0x7f404819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f404810c220 con 0x7f4048100700 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f404810c0a0 con 0x7f40481022b0 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f404e113640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f404810c3a0 con 0x7f40481018f0 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f40477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f40477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37036/0 (socket says 172.21.15.77:37036) 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f40477fe640 1 -- 172.21.15.77:0/943994127 learned_addr learned my addr 172.21.15.77:0/943994127 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f4046ffd640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40481018f0 0x7f4048197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.434+0000 7f4047fff640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40481022b0 0x7f404819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f40477fe640 1 -- 172.21.15.77:0/943994127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40481018f0 msgr2=0x7f4048197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f40477fe640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40481018f0 0x7f4048197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f40477fe640 1 -- 172.21.15.77:0/943994127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40481022b0 msgr2=0x7f404819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f40477fe640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40481022b0 0x7f404819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f40477fe640 1 -- 172.21.15.77:0/943994127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404819f690 con 0x7f4048100700 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f40477fe640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048197a70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4038000c00 tx=0x7f4038005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:52.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f4044ff9640 1 -- 172.21.15.77:0/943994127 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4038043020 con 0x7f4048100700 2024-03-16T09:41:52.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f4044ff9640 1 -- 172.21.15.77:0/943994127 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4038004350 con 0x7f4048100700 2024-03-16T09:41:52.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f404819f920 con 0x7f4048100700 2024-03-16T09:41:52.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f4044ff9640 1 -- 172.21.15.77:0/943994127 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4038003b80 con 0x7f4048100700 2024-03-16T09:41:52.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.435+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f404819fe70 con 0x7f4048100700 2024-03-16T09:41:52.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.436+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40481a0040 con 0x7f4048100700 2024-03-16T09:41:52.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.437+0000 7f4044ff9640 1 -- 172.21.15.77:0/943994127 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4038046020 con 0x7f4048100700 2024-03-16T09:41:52.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.437+0000 7f4044ff9640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40300a2800 0x7f40300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.438+0000 7f4046ffd640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40300a2800 0x7f40300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.438+0000 7f4044ff9640 1 -- 172.21.15.77:0/943994127 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f40380f0b70 con 0x7f4048100700 2024-03-16T09:41:52.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.438+0000 7f4046ffd640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40300a2800 0x7f40300a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4048198f80 tx=0x7f403c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:52.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.442+0000 7f4044ff9640 1 -- 172.21.15.77:0/943994127 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f403804d020 con 0x7f4048100700 2024-03-16T09:41:52.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.652+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40481a0210 con 0x7f4048100700 2024-03-16T09:41:52.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.653+0000 7f4044ff9640 1 -- 172.21.15.77:0/943994127 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f40380bdc20 con 0x7f4048100700 2024-03-16T09:41:52.656 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:52.656 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:41:52.656 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:41:52.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40300a2800 msgr2=0x7f40300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40300a2800 0x7f40300a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4048198f80 tx=0x7f403c005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:52.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 msgr2=0x7f4048197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048197a70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4038000c00 tx=0x7f4038005dc0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 shutdown_connections 2024-03-16T09:41:52.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f40300a2800 0x7f40300a4cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40481022b0 0x7f404819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40481018f0 0x7f4048197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 --2- 172.21.15.77:0/943994127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4048100700 0x7f4048197a70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 >> 172.21.15.77:0/943994127 conn(0x7f40480fbc70 msgr2=0x7f4048068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:52.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 shutdown_connections 2024-03-16T09:41:52.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.655+0000 7f404e113640 1 -- 172.21.15.77:0/943994127 wait complete. 2024-03-16T09:41:52.669 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-16T09:41:52.669 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-16T09:41:52.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T09:41:52.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 -- 172.21.15.77:0/355306616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac1018f0 msgr2=0x7fdbac101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/355306616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac1018f0 0x7fdbac101d70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009900 tx=0x7fdb9c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:52.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 -- 172.21.15.77:0/355306616 shutdown_connections 2024-03-16T09:41:52.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/355306616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbac1022b0 0x7fdbac108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/355306616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac1018f0 0x7fdbac101d70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/355306616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbac100700 0x7fdbac100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 -- 172.21.15.77:0/355306616 >> 172.21.15.77:0/355306616 conn(0x7fdbac0fbc90 msgr2=0x7fdbac0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 -- 172.21.15.77:0/355306616 shutdown_connections 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 -- 172.21.15.77:0/355306616 wait complete. 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 Processor -- start 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.770+0000 7fdbb37a9640 1 -- start start 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac100700 0x7fdbac06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbac1018f0 0x7fdbac19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbac1022b0 0x7fdbac19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10c230 con 0x7fdbac100700 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10c0b0 con 0x7fdbac1022b0 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10c3b0 con 0x7fdbac1018f0 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac100700 0x7fdbac06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac100700 0x7fdbac06a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37048/0 (socket says 172.21.15.77:37048) 2024-03-16T09:41:52.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 -- 172.21.15.77:0/2754535055 learned_addr learned my addr 172.21.15.77:0/2754535055 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb0d44640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbac1018f0 0x7fdbac19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1d46640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbac1022b0 0x7fdbac19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 -- 172.21.15.77:0/2754535055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbac1018f0 msgr2=0x7fdbac19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbac1018f0 0x7fdbac19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 -- 172.21.15.77:0/2754535055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbac1022b0 msgr2=0x7fdbac19edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbac1022b0 0x7fdbac19edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 -- 172.21.15.77:0/2754535055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac19f680 con 0x7fdbac100700 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb1545640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac100700 0x7fdbac06a9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdba800e9c0 tx=0x7fdba800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdba27fc640 1 -- 172.21.15.77:0/2754535055 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba800cd80 con 0x7fdbac100700 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdba27fc640 1 -- 172.21.15.77:0/2754535055 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdba80035f0 con 0x7fdbac100700 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdba27fc640 1 -- 172.21.15.77:0/2754535055 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8010690 con 0x7fdbac100700 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbac19f970 con 0x7fdbac100700 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.771+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbac19fe90 con 0x7fdbac100700 2024-03-16T09:41:52.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.772+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbac107b40 con 0x7fdbac100700 2024-03-16T09:41:52.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.773+0000 7fdba27fc640 1 -- 172.21.15.77:0/2754535055 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdba80037a0 con 0x7fdbac100700 2024-03-16T09:41:52.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.774+0000 7fdba27fc640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdb980a2800 0x7fdb980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:52.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.774+0000 7fdbb0d44640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdb980a2800 0x7fdb980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:52.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.774+0000 7fdba27fc640 1 -- 172.21.15.77:0/2754535055 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdba80cede0 con 0x7fdbac100700 2024-03-16T09:41:52.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.774+0000 7fdbb0d44640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdb980a2800 0x7fdb980a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009230 tx=0x7fdb9c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:52.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.778+0000 7fdba27fc640 1 -- 172.21.15.77:0/2754535055 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdba809be90 con 0x7fdbac100700 2024-03-16T09:41:52.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.951+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdbac061c20 con 0x7fdb980a2800 2024-03-16T09:41:52.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.958+0000 7fdba27fc640 1 -- 172.21.15.77:0/2754535055 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258804 (secure 0 0 0) 0x7fdbac061c20 con 0x7fdb980a2800 2024-03-16T09:41:52.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:52.966 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T09:41:52.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdb980a2800 msgr2=0x7fdb980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdb980a2800 0x7fdb980a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009230 tx=0x7fdb9c005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac100700 msgr2=0x7fdbac06a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac100700 0x7fdbac06a9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdba800e9c0 tx=0x7fdba800ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 shutdown_connections 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdb980a2800 0x7fdb980a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbac1022b0 0x7fdbac19edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbac1018f0 0x7fdbac19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 --2- 172.21.15.77:0/2754535055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac100700 0x7fdbac06a9c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 >> 172.21.15.77:0/2754535055 conn(0x7fdbac0fbc90 msgr2=0x7fdbac104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 shutdown_connections 2024-03-16T09:41:52.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:52.961+0000 7fdbb37a9640 1 -- 172.21.15.77:0/2754535055 wait complete. 2024-03-16T09:41:52.974 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:52 smithi077 ceph-mon[32171]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:41:52.974 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:52 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3354665720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.974 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:52 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1682060792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:52.974 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:52 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/943994127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:52.975 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-16T09:41:51.800371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655470,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":816,"num_read_kb":525,"num_write":352,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358300,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583524,"statfs":{"total":1151940427776,"available":1151573528576,"internally_reserved":0,"allocated":13627392,"data_stored":7488704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":986,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003299"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.812919+0000","last_change":"2024-03-16T09:39:56.184389+0000","last_active":"2024-03-16T09:41:27.812919+0000","last_peered":"2024-03-16T09:41:27.812919+0000","last_clean":"2024-03-16T09:41:27.812919+0000","last_became_active":"2024-03-16T09:39:56.184129+0000","last_became_peered":"2024-03-16T09:39:56.184129+0000","last_unstale":"2024-03-16T09:41:27.812919+0000","last_undegraded":"2024-03-16T09:41:27.812919+0000","last_fullsized":"2024-03-16T09:41:27.812919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:33.786788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.800995+0000","last_change":"2024-03-16T09:39:56.193451+0000","last_active":"2024-03-16T09:41:27.800995+0000","last_peered":"2024-03-16T09:41:27.800995+0000","last_clean":"2024-03-16T09:41:27.800995+0000","last_became_active":"2024-03-16T09:39:56.187381+0000","last_became_peered":"2024-03-16T09:39:56.187381+0000","last_unstale":"2024-03-16T09:41:27.800995+0000","last_undegraded":"2024-03-16T09:41:27.800995+0000","last_fullsized":"2024-03-16T09:41:27.800995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:51.915654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816523+0000","last_change":"2024-03-16T09:39:56.183785+0000","last_active":"2024-03-16T09:41:27.816523+0000","last_peered":"2024-03-16T09:41:27.816523+0000","last_clean":"2024-03-16T09:41:27.816523+0000","last_became_active":"2024-03-16T09:39:56.183421+0000","last_became_peered":"2024-03-16T09:39:56.183421+0000","last_unstale":"2024-03-16T09:41:27.816523+0000","last_undegraded":"2024-03-16T09:41:27.816523+0000","last_fullsized":"2024-03-16T09:41:27.816523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:47.259139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806416+0000","last_change":"2024-03-16T09:39:56.195743+0000","last_active":"2024-03-16T09:41:27.806416+0000","last_peered":"2024-03-16T09:41:27.806416+0000","last_clean":"2024-03-16T09:41:27.806416+0000","last_became_active":"2024-03-16T09:39:56.194818+0000","last_became_peered":"2024-03-16T09:39:56.194818+0000","last_unstale":"2024-03-16T09:41:27.806416+0000","last_undegraded":"2024-03-16T09:41:27.806416+0000","last_fullsized":"2024-03-16T09:41:27.806416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:39:43.123879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.809286+0000","last_change":"2024-03-16T09:39:56.184970+0000","last_active":"2024-03-16T09:41:27.809286+0000","last_peered":"2024-03-16T09:41:27.809286+0000","last_clean":"2024-03-16T09:41:27.809286+0000","last_became_active":"2024-03-16T09:39:56.184632+0000","last_became_peered":"2024-03-16T09:39:56.184632+0000","last_unstale":"2024-03-16T09:41:27.809286+0000","last_undegraded":"2024-03-16T09:41:27.809286+0000","last_fullsized":"2024-03-16T09:41:27.809286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:46:51.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804440+0000","last_change":"2024-03-16T09:39:56.199688+0000","last_active":"2024-03-16T09:41:27.804440+0000","last_peered":"2024-03-16T09:41:27.804440+0000","last_clean":"2024-03-16T09:41:27.804440+0000","last_became_active":"2024-03-16T09:39:56.199094+0000","last_became_peered":"2024-03-16T09:39:56.199094+0000","last_unstale":"2024-03-16T09:41:27.804440+0000","last_undegraded":"2024-03-16T09:41:27.804440+0000","last_fullsized":"2024-03-16T09:41:27.804440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:10.855504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.809387+0000","last_change":"2024-03-16T09:39:56.180816+0000","last_active":"2024-03-16T09:41:27.809387+0000","last_peered":"2024-03-16T09:41:27.809387+0000","last_clean":"2024-03-16T09:41:27.809387+0000","last_became_active":"2024-03-16T09:39:56.180499+0000","last_became_peered":"2024-03-16T09:39:56.180499+0000","last_unstale":"2024-03-16T09:41:27.809387+0000","last_undegraded":"2024-03-16T09:41:27.809387+0000","last_fullsized":"2024-03-16T09:41:27.809387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:32:09.444494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802950+0000","last_change":"2024-03-16T09:39:56.189960+0000","last_active":"2024-03-16T09:41:27.802950+0000","last_peered":"2024-03-16T09:41:27.802950+0000","last_clean":"2024-03-16T09:41:27.802950+0000","last_became_active":"2024-03-16T09:39:56.189609+0000","last_became_peered":"2024-03-16T09:39:56.189609+0000","last_unstale":"2024-03-16T09:41:27.802950+0000","last_undegraded":"2024-03-16T09:41:27.802950+0000","last_fullsized":"2024-03-16T09:41:27.802950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:50.219703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816982+0000","last_change":"2024-03-16T09:40:58.867996+0000","last_active":"2024-03-16T09:41:27.816982+0000","last_peered":"2024-03-16T09:41:27.816982+0000","last_clean":"2024-03-16T09:41:27.816982+0000","last_became_active":"2024-03-16T09:40:58.867619+0000","last_became_peered":"2024-03-16T09:40:58.867619+0000","last_unstale":"2024-03-16T09:41:27.816982+0000","last_undegraded":"2024-03-16T09:41:27.816982+0000","last_fullsized":"2024-03-16T09:41:27.816982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:36.972983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804258+0000","last_change":"2024-03-16T09:39:56.201584+0000","last_active":"2024-03-16T09:41:27.804258+0000","last_peered":"2024-03-16T09:41:27.804258+0000","last_clean":"2024-03-16T09:41:27.804258+0000","last_became_active":"2024-03-16T09:39:56.201320+0000","last_became_peered":"2024-03-16T09:39:56.201320+0000","last_unstale":"2024-03-16T09:41:27.804258+0000","last_undegraded":"2024-03-16T09:41:27.804258+0000","last_fullsized":"2024-03-16T09:41:27.804258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:18:00.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817302+0000","last_change":"2024-03-16T09:39:56.191125+0000","last_active":"2024-03-16T09:41:27.817302+0000","last_peered":"2024-03-16T09:41:27.817302+0000","last_clean":"2024-03-16T09:41:27.817302+0000","last_became_active":"2024-03-16T09:39:56.190795+0000","last_became_peered":"2024-03-16T09:39:56.190795+0000","last_unstale":"2024-03-16T09:41:27.817302+0000","last_undegraded":"2024-03-16T09:41:27.817302+0000","last_fullsized":"2024-03-16T09:41:27.817302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:44:10.894995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805793+0000","last_change":"2024-03-16T09:39:56.195652+0000","last_active":"2024-03-16T09:41:27.805793+0000","last_peered":"2024-03-16T09:41:27.805793+0000","last_clean":"2024-03-16T09:41:27.805793+0000","last_became_active":"2024-03-16T09:39:56.195270+0000","last_became_peered":"2024-03-16T09:39:56.195270+0000","last_unstale":"2024-03-16T09:41:27.805793+0000","last_undegraded":"2024-03-16T09:41:27.805793+0000","last_fullsized":"2024-03-16T09:41:27.805793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:03:41.546991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810598+0000","last_change":"2024-03-16T09:39:56.184622+0000","last_active":"2024-03-16T09:41:27.810598+0000","last_peered":"2024-03-16T09:41:27.810598+0000","last_clean":"2024-03-16T09:41:27.810598+0000","last_became_active":"2024-03-16T09:39:56.183532+0000","last_became_peered":"2024-03-16T09:39:56.183532+0000","last_unstale":"2024-03-16T09:41:27.810598+0000","last_undegraded":"2024-03-16T09:41:27.810598+0000","last_fullsized":"2024-03-16T09:41:27.810598+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:11:24.574088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808922+0000","last_change":"2024-03-16T09:39:56.184393+0000","last_active":"2024-03-16T09:41:27.808922+0000","last_peered":"2024-03-16T09:41:27.808922+0000","last_clean":"2024-03-16T09:41:27.808922+0000","last_became_active":"2024-03-16T09:39:56.183783+0000","last_became_peered":"2024-03-16T09:39:56.183783+0000","last_unstale":"2024-03-16T09:41:27.808922+0000","last_undegraded":"2024-03-16T09:41:27.808922+0000","last_fullsized":"2024-03-16T09:41:27.808922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:16:22.131116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811183+0000","last_change":"2024-03-16T09:39:56.197365+0000","last_active":"2024-03-16T09:41:27.811183+0000","last_peered":"2024-03-16T09:41:27.811183+0000","last_clean":"2024-03-16T09:41:27.811183+0000","last_became_active":"2024-03-16T09:39:56.197020+0000","last_became_peered":"2024-03-16T09:39:56.197020+0000","last_unstale":"2024-03-16T09:41:27.811183+0000","last_undegraded":"2024-03-16T09:41:27.811183+0000","last_fullsized":"2024-03-16T09:41:27.811183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:29.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817996+0000","last_change":"2024-03-16T09:39:56.185513+0000","last_active":"2024-03-16T09:41:27.817996+0000","last_peered":"2024-03-16T09:41:27.817996+0000","last_clean":"2024-03-16T09:41:27.817996+0000","last_became_active":"2024-03-16T09:39:56.184384+0000","last_became_peered":"2024-03-16T09:39:56.184384+0000","last_unstale":"2024-03-16T09:41:27.817996+0000","last_undegraded":"2024-03-16T09:41:27.817996+0000","last_fullsized":"2024-03-16T09:41:27.817996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:29.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.813059+0000","last_change":"2024-03-16T09:39:56.184536+0000","last_active":"2024-03-16T09:41:27.813059+0000","last_peered":"2024-03-16T09:41:27.813059+0000","last_clean":"2024-03-16T09:41:27.813059+0000","last_became_active":"2024-03-16T09:39:56.184208+0000","last_became_peered":"2024-03-16T09:39:56.184208+0000","last_unstale":"2024-03-16T09:41:27.813059+0000","last_undegraded":"2024-03-16T09:41:27.813059+0000","last_fullsized":"2024-03-16T09:41:27.813059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:14.025625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.796187+0000","last_change":"2024-03-16T09:39:56.184331+0000","last_active":"2024-03-16T09:41:27.796187+0000","last_peered":"2024-03-16T09:41:27.796187+0000","last_clean":"2024-03-16T09:41:27.796187+0000","last_became_active":"2024-03-16T09:39:56.183919+0000","last_became_peered":"2024-03-16T09:39:56.183919+0000","last_unstale":"2024-03-16T09:41:27.796187+0000","last_undegraded":"2024-03-16T09:41:27.796187+0000","last_fullsized":"2024-03-16T09:41:27.796187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:59.692854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802842+0000","last_change":"2024-03-16T09:39:56.185607+0000","last_active":"2024-03-16T09:41:27.802842+0000","last_peered":"2024-03-16T09:41:27.802842+0000","last_clean":"2024-03-16T09:41:27.802842+0000","last_became_active":"2024-03-16T09:39:56.185278+0000","last_became_peered":"2024-03-16T09:39:56.185278+0000","last_unstale":"2024-03-16T09:41:27.802842+0000","last_undegraded":"2024-03-16T09:41:27.802842+0000","last_fullsized":"2024-03-16T09:41:27.802842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:20.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860105+0000","last_change":"2024-03-16T09:39:55.180852+0000","last_active":"2024-03-16T09:40:58.860105+0000","last_peered":"2024-03-16T09:40:58.860105+0000","last_clean":"2024-03-16T09:40:58.860105+0000","last_became_active":"2024-03-16T09:39:55.180496+0000","last_became_peered":"2024-03-16T09:39:55.180496+0000","last_unstale":"2024-03-16T09:40:58.860105+0000","last_undegraded":"2024-03-16T09:40:58.860105+0000","last_fullsized":"2024-03-16T09:40:58.860105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:15:28.548458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805721+0000","last_change":"2024-03-16T09:39:56.190110+0000","last_active":"2024-03-16T09:41:27.805721+0000","last_peered":"2024-03-16T09:41:27.805721+0000","last_clean":"2024-03-16T09:41:27.805721+0000","last_became_active":"2024-03-16T09:39:56.189711+0000","last_became_peered":"2024-03-16T09:39:56.189711+0000","last_unstale":"2024-03-16T09:41:27.805721+0000","last_undegraded":"2024-03-16T09:41:27.805721+0000","last_fullsized":"2024-03-16T09:41:27.805721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:17.353457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871621+0000","last_change":"2024-03-16T09:39:55.171142+0000","last_active":"2024-03-16T09:40:58.871621+0000","last_peered":"2024-03-16T09:40:58.871621+0000","last_clean":"2024-03-16T09:40:58.871621+0000","last_became_active":"2024-03-16T09:39:55.170793+0000","last_became_peered":"2024-03-16T09:39:55.170793+0000","last_unstale":"2024-03-16T09:40:58.871621+0000","last_undegraded":"2024-03-16T09:40:58.871621+0000","last_fullsized":"2024-03-16T09:40:58.871621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:36.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816268+0000","last_change":"2024-03-16T09:39:56.186802+0000","last_active":"2024-03-16T09:41:27.816268+0000","last_peered":"2024-03-16T09:41:27.816268+0000","last_clean":"2024-03-16T09:41:27.816268+0000","last_became_active":"2024-03-16T09:39:56.186398+0000","last_became_peered":"2024-03-16T09:39:56.186398+0000","last_unstale":"2024-03-16T09:41:27.816268+0000","last_undegraded":"2024-03-16T09:41:27.816268+0000","last_fullsized":"2024-03-16T09:41:27.816268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:05:56.209864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864171+0000","last_change":"2024-03-16T09:39:55.181894+0000","last_active":"2024-03-16T09:40:58.864171+0000","last_peered":"2024-03-16T09:40:58.864171+0000","last_clean":"2024-03-16T09:40:58.864171+0000","last_became_active":"2024-03-16T09:39:55.179434+0000","last_became_peered":"2024-03-16T09:39:55.179434+0000","last_unstale":"2024-03-16T09:40:58.864171+0000","last_undegraded":"2024-03-16T09:40:58.864171+0000","last_fullsized":"2024-03-16T09:40:58.864171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:16.153928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818117+0000","last_change":"2024-03-16T09:39:56.188464+0000","last_active":"2024-03-16T09:41:27.818117+0000","last_peered":"2024-03-16T09:41:27.818117+0000","last_clean":"2024-03-16T09:41:27.818117+0000","last_became_active":"2024-03-16T09:39:56.188125+0000","last_became_peered":"2024-03-16T09:39:56.188125+0000","last_unstale":"2024-03-16T09:41:27.818117+0000","last_undegraded":"2024-03-16T09:41:27.818117+0000","last_fullsized":"2024-03-16T09:41:27.818117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:46:49.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863856+0000","last_change":"2024-03-16T09:39:55.179379+0000","last_active":"2024-03-16T09:40:58.863856+0000","last_peered":"2024-03-16T09:40:58.863856+0000","last_clean":"2024-03-16T09:40:58.863856+0000","last_became_active":"2024-03-16T09:39:55.178755+0000","last_became_peered":"2024-03-16T09:39:55.178755+0000","last_unstale":"2024-03-16T09:40:58.863856+0000","last_undegraded":"2024-03-16T09:40:58.863856+0000","last_fullsized":"2024-03-16T09:40:58.863856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:07.131417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805630+0000","last_change":"2024-03-16T09:39:56.196655+0000","last_active":"2024-03-16T09:41:27.805630+0000","last_peered":"2024-03-16T09:41:27.805630+0000","last_clean":"2024-03-16T09:41:27.805630+0000","last_became_active":"2024-03-16T09:39:56.196344+0000","last_became_peered":"2024-03-16T09:39:56.196344+0000","last_unstale":"2024-03-16T09:41:27.805630+0000","last_undegraded":"2024-03-16T09:41:27.805630+0000","last_fullsized":"2024-03-16T09:41:27.805630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:12:03.920319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":134,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:45.842807+0000","last_change":"2024-03-16T09:39:55.182012+0000","last_active":"2024-03-16T09:41:45.842807+0000","last_peered":"2024-03-16T09:41:45.842807+0000","last_clean":"2024-03-16T09:41:45.842807+0000","last_became_active":"2024-03-16T09:39:55.178651+0000","last_became_peered":"2024-03-16T09:39:55.178651+0000","last_unstale":"2024-03-16T09:41:45.842807+0000","last_undegraded":"2024-03-16T09:41:45.842807+0000","last_fullsized":"2024-03-16T09:41:45.842807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:50:19.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.795850+0000","last_change":"2024-03-16T09:39:56.179645+0000","last_active":"2024-03-16T09:41:27.795850+0000","last_peered":"2024-03-16T09:41:27.795850+0000","last_clean":"2024-03-16T09:41:27.795850+0000","last_became_active":"2024-03-16T09:39:56.179257+0000","last_became_peered":"2024-03-16T09:39:56.179257+0000","last_unstale":"2024-03-16T09:41:27.795850+0000","last_undegraded":"2024-03-16T09:41:27.795850+0000","last_fullsized":"2024-03-16T09:41:27.795850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:30:09.104630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.218444+0000","last_change":"2024-03-16T09:39:55.178257+0000","last_active":"2024-03-16T09:40:59.218444+0000","last_peered":"2024-03-16T09:40:59.218444+0000","last_clean":"2024-03-16T09:40:59.218444+0000","last_became_active":"2024-03-16T09:39:55.177984+0000","last_became_peered":"2024-03-16T09:39:55.177984+0000","last_unstale":"2024-03-16T09:40:59.218444+0000","last_undegraded":"2024-03-16T09:40:59.218444+0000","last_fullsized":"2024-03-16T09:40:59.218444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.385624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804510+0000","last_change":"2024-03-16T09:39:56.196821+0000","last_active":"2024-03-16T09:41:27.804510+0000","last_peered":"2024-03-16T09:41:27.804510+0000","last_clean":"2024-03-16T09:41:27.804510+0000","last_became_active":"2024-03-16T09:39:56.196522+0000","last_became_peered":"2024-03-16T09:39:56.196522+0000","last_unstale":"2024-03-16T09:41:27.804510+0000","last_undegraded":"2024-03-16T09:41:27.804510+0000","last_fullsized":"2024-03-16T09:41:27.804510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:31.103530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865998+0000","last_change":"2024-03-16T09:39:55.170290+0000","last_active":"2024-03-16T09:40:58.865998+0000","last_peered":"2024-03-16T09:40:58.865998+0000","last_clean":"2024-03-16T09:40:58.865998+0000","last_became_active":"2024-03-16T09:39:55.169956+0000","last_became_peered":"2024-03-16T09:39:55.169956+0000","last_unstale":"2024-03-16T09:40:58.865998+0000","last_undegraded":"2024-03-16T09:40:58.865998+0000","last_fullsized":"2024-03-16T09:40:58.865998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:01:31.606917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.801687+0000","last_change":"2024-03-16T09:39:56.189155+0000","last_active":"2024-03-16T09:41:27.801687+0000","last_peered":"2024-03-16T09:41:27.801687+0000","last_clean":"2024-03-16T09:41:27.801687+0000","last_became_active":"2024-03-16T09:39:56.188827+0000","last_became_peered":"2024-03-16T09:39:56.188827+0000","last_unstale":"2024-03-16T09:41:27.801687+0000","last_undegraded":"2024-03-16T09:41:27.801687+0000","last_fullsized":"2024-03-16T09:41:27.801687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:54.370754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863093+0000","last_change":"2024-03-16T09:39:55.181886+0000","last_active":"2024-03-16T09:40:58.863093+0000","last_peered":"2024-03-16T09:40:58.863093+0000","last_clean":"2024-03-16T09:40:58.863093+0000","last_became_active":"2024-03-16T09:39:55.179836+0000","last_became_peered":"2024-03-16T09:39:55.179836+0000","last_unstale":"2024-03-16T09:40:58.863093+0000","last_undegraded":"2024-03-16T09:40:58.863093+0000","last_fullsized":"2024-03-16T09:40:58.863093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:52:38.772861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.795474+0000","last_change":"2024-03-16T09:39:56.185892+0000","last_active":"2024-03-16T09:41:27.795474+0000","last_peered":"2024-03-16T09:41:27.795474+0000","last_clean":"2024-03-16T09:41:27.795474+0000","last_became_active":"2024-03-16T09:39:56.185459+0000","last_became_peered":"2024-03-16T09:39:56.185459+0000","last_unstale":"2024-03-16T09:41:27.795474+0000","last_undegraded":"2024-03-16T09:41:27.795474+0000","last_fullsized":"2024-03-16T09:41:27.795474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:10.421054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":134,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:48.845916+0000","last_change":"2024-03-16T09:39:55.177607+0000","last_active":"2024-03-16T09:41:48.845916+0000","last_peered":"2024-03-16T09:41:48.845916+0000","last_clean":"2024-03-16T09:41:48.845916+0000","last_became_active":"2024-03-16T09:39:55.177265+0000","last_became_peered":"2024-03-16T09:39:55.177265+0000","last_unstale":"2024-03-16T09:41:48.845916+0000","last_undegraded":"2024-03-16T09:41:48.845916+0000","last_fullsized":"2024-03-16T09:41:48.845916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:47.849041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818050+0000","last_change":"2024-03-16T09:39:56.189172+0000","last_active":"2024-03-16T09:41:27.818050+0000","last_peered":"2024-03-16T09:41:27.818050+0000","last_clean":"2024-03-16T09:41:27.818050+0000","last_became_active":"2024-03-16T09:39:56.188798+0000","last_became_peered":"2024-03-16T09:39:56.188798+0000","last_unstale":"2024-03-16T09:41:27.818050+0000","last_undegraded":"2024-03-16T09:41:27.818050+0000","last_fullsized":"2024-03-16T09:41:27.818050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:02:11.119293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860780+0000","last_change":"2024-03-16T09:39:55.180369+0000","last_active":"2024-03-16T09:40:58.860780+0000","last_peered":"2024-03-16T09:40:58.860780+0000","last_clean":"2024-03-16T09:40:58.860780+0000","last_became_active":"2024-03-16T09:39:55.180053+0000","last_became_peered":"2024-03-16T09:39:55.180053+0000","last_unstale":"2024-03-16T09:40:58.860780+0000","last_undegraded":"2024-03-16T09:40:58.860780+0000","last_fullsized":"2024-03-16T09:40:58.860780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:44.990028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811041+0000","last_change":"2024-03-16T09:39:56.198454+0000","last_active":"2024-03-16T09:41:27.811041+0000","last_peered":"2024-03-16T09:41:27.811041+0000","last_clean":"2024-03-16T09:41:27.811041+0000","last_became_active":"2024-03-16T09:39:56.198097+0000","last_became_peered":"2024-03-16T09:39:56.198097+0000","last_unstale":"2024-03-16T09:41:27.811041+0000","last_undegraded":"2024-03-16T09:41:27.811041+0000","last_fullsized":"2024-03-16T09:41:27.811041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:12.494363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863700+0000","last_change":"2024-03-16T09:39:55.174221+0000","last_active":"2024-03-16T09:40:58.863700+0000","last_peered":"2024-03-16T09:40:58.863700+0000","last_clean":"2024-03-16T09:40:58.863700+0000","last_became_active":"2024-03-16T09:39:55.173920+0000","last_became_peered":"2024-03-16T09:39:55.173920+0000","last_unstale":"2024-03-16T09:40:58.863700+0000","last_undegraded":"2024-03-16T09:40:58.863700+0000","last_fullsized":"2024-03-16T09:40:58.863700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:25.108414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816889+0000","last_change":"2024-03-16T09:39:56.196849+0000","last_active":"2024-03-16T09:41:27.816889+0000","last_peered":"2024-03-16T09:41:27.816889+0000","last_clean":"2024-03-16T09:41:27.816889+0000","last_became_active":"2024-03-16T09:39:56.196515+0000","last_became_peered":"2024-03-16T09:39:56.196515+0000","last_unstale":"2024-03-16T09:41:27.816889+0000","last_undegraded":"2024-03-16T09:41:27.816889+0000","last_fullsized":"2024-03-16T09:41:27.816889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:03.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":285,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:49.846725+0000","last_change":"2024-03-16T09:39:55.177788+0000","last_active":"2024-03-16T09:41:49.846725+0000","last_peered":"2024-03-16T09:41:49.846725+0000","last_clean":"2024-03-16T09:41:49.846725+0000","last_became_active":"2024-03-16T09:39:55.177443+0000","last_became_peered":"2024-03-16T09:39:55.177443+0000","last_unstale":"2024-03-16T09:41:49.846725+0000","last_undegraded":"2024-03-16T09:41:49.846725+0000","last_fullsized":"2024-03-16T09:41:49.846725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:39.221712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60765,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":232,"num_read_kb":175,"num_write":35,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.807296+0000","last_change":"2024-03-16T09:39:56.178976+0000","last_active":"2024-03-16T09:41:27.807296+0000","last_peered":"2024-03-16T09:41:27.807296+0000","last_clean":"2024-03-16T09:41:27.807296+0000","last_became_active":"2024-03-16T09:39:56.178455+0000","last_became_peered":"2024-03-16T09:39:56.178455+0000","last_unstale":"2024-03-16T09:41:27.807296+0000","last_undegraded":"2024-03-16T09:41:27.807296+0000","last_fullsized":"2024-03-16T09:41:27.807296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:52:08.064792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865191+0000","last_change":"2024-03-16T09:39:55.182579+0000","last_active":"2024-03-16T09:40:58.865191+0000","last_peered":"2024-03-16T09:40:58.865191+0000","last_clean":"2024-03-16T09:40:58.865191+0000","last_became_active":"2024-03-16T09:39:55.181298+0000","last_became_peered":"2024-03-16T09:39:55.181298+0000","last_unstale":"2024-03-16T09:40:58.865191+0000","last_undegraded":"2024-03-16T09:40:58.865191+0000","last_fullsized":"2024-03-16T09:40:58.865191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:35.768812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814563+0000","last_change":"2024-03-16T09:39:56.188519+0000","last_active":"2024-03-16T09:41:27.814563+0000","last_peered":"2024-03-16T09:41:27.814563+0000","last_clean":"2024-03-16T09:41:27.814563+0000","last_became_active":"2024-03-16T09:39:56.188031+0000","last_became_peered":"2024-03-16T09:39:56.188031+0000","last_unstale":"2024-03-16T09:41:27.814563+0000","last_undegraded":"2024-03-16T09:41:27.814563+0000","last_fullsized":"2024-03-16T09:41:27.814563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:20:31.765657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864304+0000","last_change":"2024-03-16T09:39:55.177523+0000","last_active":"2024-03-16T09:40:58.864304+0000","last_peered":"2024-03-16T09:40:58.864304+0000","last_clean":"2024-03-16T09:40:58.864304+0000","last_became_active":"2024-03-16T09:39:55.177143+0000","last_became_peered":"2024-03-16T09:39:55.177143+0000","last_unstale":"2024-03-16T09:40:58.864304+0000","last_undegraded":"2024-03-16T09:40:58.864304+0000","last_fullsized":"2024-03-16T09:40:58.864304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:05.357611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.813082+0000","last_change":"2024-03-16T09:40:58.867151+0000","last_active":"2024-03-16T09:41:27.813082+0000","last_peered":"2024-03-16T09:41:27.813082+0000","last_clean":"2024-03-16T09:41:27.813082+0000","last_became_active":"2024-03-16T09:40:58.866756+0000","last_became_peered":"2024-03-16T09:40:58.866756+0000","last_unstale":"2024-03-16T09:41:27.813082+0000","last_undegraded":"2024-03-16T09:41:27.813082+0000","last_fullsized":"2024-03-16T09:41:27.813082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:53.323964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:47.866162+0000","last_change":"2024-03-16T09:39:55.175419+0000","last_active":"2024-03-16T09:41:47.866162+0000","last_peered":"2024-03-16T09:41:47.866162+0000","last_clean":"2024-03-16T09:41:47.866162+0000","last_became_active":"2024-03-16T09:39:55.175126+0000","last_became_peered":"2024-03-16T09:39:55.175126+0000","last_unstale":"2024-03-16T09:41:47.866162+0000","last_undegraded":"2024-03-16T09:41:47.866162+0000","last_fullsized":"2024-03-16T09:41:47.866162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:11.696911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.812473+0000","last_change":"2024-03-16T09:39:56.176714+0000","last_active":"2024-03-16T09:41:27.812473+0000","last_peered":"2024-03-16T09:41:27.812473+0000","last_clean":"2024-03-16T09:41:27.812473+0000","last_became_active":"2024-03-16T09:39:56.176245+0000","last_became_peered":"2024-03-16T09:39:56.176245+0000","last_unstale":"2024-03-16T09:41:27.812473+0000","last_undegraded":"2024-03-16T09:41:27.812473+0000","last_fullsized":"2024-03-16T09:41:27.812473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:32.430207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863725+0000","last_change":"2024-03-16T09:39:55.181486+0000","last_active":"2024-03-16T09:40:58.863725+0000","last_peered":"2024-03-16T09:40:58.863725+0000","last_clean":"2024-03-16T09:40:58.863725+0000","last_became_active":"2024-03-16T09:39:55.180128+0000","last_became_peered":"2024-03-16T09:39:55.180128+0000","last_unstale":"2024-03-16T09:40:58.863725+0000","last_undegraded":"2024-03-16T09:40:58.863725+0000","last_fullsized":"2024-03-16T09:40:58.863725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:45.323870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803486+0000","last_change":"2024-03-16T09:39:56.196692+0000","last_active":"2024-03-16T09:41:27.803486+0000","last_peered":"2024-03-16T09:41:27.803486+0000","last_clean":"2024-03-16T09:41:27.803486+0000","last_became_active":"2024-03-16T09:39:56.196403+0000","last_became_peered":"2024-03-16T09:39:56.196403+0000","last_unstale":"2024-03-16T09:41:27.803486+0000","last_undegraded":"2024-03-16T09:41:27.803486+0000","last_fullsized":"2024-03-16T09:41:27.803486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:00:29.426476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871091+0000","last_change":"2024-03-16T09:40:58.870587+0000","last_active":"2024-03-16T09:40:58.871091+0000","last_peered":"2024-03-16T09:40:58.871091+0000","last_clean":"2024-03-16T09:40:58.871091+0000","last_became_active":"2024-03-16T09:40:58.864977+0000","last_became_peered":"2024-03-16T09:40:58.864977+0000","last_unstale":"2024-03-16T09:40:58.871091+0000","last_undegraded":"2024-03-16T09:40:58.871091+0000","last_fullsized":"2024-03-16T09:40:58.871091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:18.189307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.798760+0000","last_change":"2024-03-16T09:40:58.866074+0000","last_active":"2024-03-16T09:41:27.798760+0000","last_peered":"2024-03-16T09:41:27.798760+0000","last_clean":"2024-03-16T09:41:27.798760+0000","last_became_active":"2024-03-16T09:40:58.865711+0000","last_became_peered":"2024-03-16T09:40:58.865711+0000","last_unstale":"2024-03-16T09:41:27.798760+0000","last_undegraded":"2024-03-16T09:41:27.798760+0000","last_fullsized":"2024-03-16T09:41:27.798760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:57.137802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.859988+0000","last_change":"2024-03-16T09:39:55.175666+0000","last_active":"2024-03-16T09:40:58.859988+0000","last_peered":"2024-03-16T09:40:58.859988+0000","last_clean":"2024-03-16T09:40:58.859988+0000","last_became_active":"2024-03-16T09:39:55.175337+0000","last_became_peered":"2024-03-16T09:39:55.175337+0000","last_unstale":"2024-03-16T09:40:58.859988+0000","last_undegraded":"2024-03-16T09:40:58.859988+0000","last_fullsized":"2024-03-16T09:40:58.859988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:45:13.935669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810754+0000","last_change":"2024-03-16T09:39:56.184878+0000","last_active":"2024-03-16T09:41:27.810754+0000","last_peered":"2024-03-16T09:41:27.810754+0000","last_clean":"2024-03-16T09:41:27.810754+0000","last_became_active":"2024-03-16T09:39:56.184480+0000","last_became_peered":"2024-03-16T09:39:56.184480+0000","last_unstale":"2024-03-16T09:41:27.810754+0000","last_undegraded":"2024-03-16T09:41:27.810754+0000","last_fullsized":"2024-03-16T09:41:27.810754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:40:01.286993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860584+0000","last_change":"2024-03-16T09:39:55.179189+0000","last_active":"2024-03-16T09:40:58.860584+0000","last_peered":"2024-03-16T09:40:58.860584+0000","last_clean":"2024-03-16T09:40:58.860584+0000","last_became_active":"2024-03-16T09:39:55.178859+0000","last_became_peered":"2024-03-16T09:39:55.178859+0000","last_unstale":"2024-03-16T09:40:58.860584+0000","last_undegraded":"2024-03-16T09:40:58.860584+0000","last_fullsized":"2024-03-16T09:40:58.860584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:54:15.498386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.788321+0000","last_change":"2024-03-16T09:39:56.200665+0000","last_active":"2024-03-16T09:41:27.788321+0000","last_peered":"2024-03-16T09:41:27.788321+0000","last_clean":"2024-03-16T09:41:27.788321+0000","last_became_active":"2024-03-16T09:39:56.200326+0000","last_became_peered":"2024-03-16T09:39:56.200326+0000","last_unstale":"2024-03-16T09:41:27.788321+0000","last_undegraded":"2024-03-16T09:41:27.788321+0000","last_fullsized":"2024-03-16T09:41:27.788321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:34.778630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806467+0000","last_change":"2024-03-16T09:39:56.182670+0000","last_active":"2024-03-16T09:41:27.806467+0000","last_peered":"2024-03-16T09:41:27.806467+0000","last_clean":"2024-03-16T09:41:27.806467+0000","last_became_active":"2024-03-16T09:39:56.182009+0000","last_became_peered":"2024-03-16T09:39:56.182009+0000","last_unstale":"2024-03-16T09:41:27.806467+0000","last_undegraded":"2024-03-16T09:41:27.806467+0000","last_fullsized":"2024-03-16T09:41:27.806467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:28:03.106071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:49.869064+0000","last_change":"2024-03-16T09:39:55.171609+0000","last_active":"2024-03-16T09:41:49.869064+0000","last_peered":"2024-03-16T09:41:49.869064+0000","last_clean":"2024-03-16T09:41:49.869064+0000","last_became_active":"2024-03-16T09:39:55.171301+0000","last_became_peered":"2024-03-16T09:39:55.171301+0000","last_unstale":"2024-03-16T09:41:49.869064+0000","last_undegraded":"2024-03-16T09:41:49.869064+0000","last_fullsized":"2024-03-16T09:41:49.869064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:28.265342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.787638+0000","last_change":"2024-03-16T09:40:58.865311+0000","last_active":"2024-03-16T09:41:27.787638+0000","last_peered":"2024-03-16T09:41:27.787638+0000","last_clean":"2024-03-16T09:41:27.787638+0000","last_became_active":"2024-03-16T09:40:58.864918+0000","last_became_peered":"2024-03-16T09:40:58.864918+0000","last_unstale":"2024-03-16T09:41:27.787638+0000","last_undegraded":"2024-03-16T09:41:27.787638+0000","last_fullsized":"2024-03-16T09:41:27.787638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863236+0000","last_change":"2024-03-16T09:39:55.172824+0000","last_active":"2024-03-16T09:40:58.863236+0000","last_peered":"2024-03-16T09:40:58.863236+0000","last_clean":"2024-03-16T09:40:58.863236+0000","last_became_active":"2024-03-16T09:39:55.172493+0000","last_became_peered":"2024-03-16T09:39:55.172493+0000","last_unstale":"2024-03-16T09:40:58.863236+0000","last_undegraded":"2024-03-16T09:40:58.863236+0000","last_fullsized":"2024-03-16T09:40:58.863236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:39.188176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":346,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860504+0000","last_change":"2024-03-16T09:38:43.693644+0000","last_active":"2024-03-16T09:40:58.860504+0000","last_peered":"2024-03-16T09:40:58.860504+0000","last_clean":"2024-03-16T09:40:58.860504+0000","last_became_active":"2024-03-16T09:38:43.693202+0000","last_became_peered":"2024-03-16T09:38:43.693202+0000","last_unstale":"2024-03-16T09:40:58.860504+0000","last_undegraded":"2024-03-16T09:40:58.860504+0000","last_fullsized":"2024-03-16T09:40:58.860504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_clean_scrub_stamp":"2024-03-16T09:35:24.153508+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:43:06.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810201+0000","last_change":"2024-03-16T09:39:56.197109+0000","last_active":"2024-03-16T09:41:27.810201+0000","last_peered":"2024-03-16T09:41:27.810201+0000","last_clean":"2024-03-16T09:41:27.810201+0000","last_became_active":"2024-03-16T09:39:56.196775+0000","last_became_peered":"2024-03-16T09:39:56.196775+0000","last_unstale":"2024-03-16T09:41:27.810201+0000","last_undegraded":"2024-03-16T09:41:27.810201+0000","last_fullsized":"2024-03-16T09:41:27.810201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:29:10.401423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860552+0000","last_change":"2024-03-16T09:39:55.175744+0000","last_active":"2024-03-16T09:40:58.860552+0000","last_peered":"2024-03-16T09:40:58.860552+0000","last_clean":"2024-03-16T09:40:58.860552+0000","last_became_active":"2024-03-16T09:39:55.175445+0000","last_became_peered":"2024-03-16T09:39:55.175445+0000","last_unstale":"2024-03-16T09:40:58.860552+0000","last_undegraded":"2024-03-16T09:40:58.860552+0000","last_fullsized":"2024-03-16T09:40:58.860552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:39.494340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864068+0000","last_change":"2024-03-16T09:39:55.183441+0000","last_active":"2024-03-16T09:40:58.864068+0000","last_peered":"2024-03-16T09:40:58.864068+0000","last_clean":"2024-03-16T09:40:58.864068+0000","last_became_active":"2024-03-16T09:39:55.183115+0000","last_became_peered":"2024-03-16T09:39:55.183115+0000","last_unstale":"2024-03-16T09:40:58.864068+0000","last_undegraded":"2024-03-16T09:40:58.864068+0000","last_fullsized":"2024-03-16T09:40:58.864068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:55:31.525201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804957+0000","last_change":"2024-03-16T09:39:56.179205+0000","last_active":"2024-03-16T09:41:27.804957+0000","last_peered":"2024-03-16T09:41:27.804957+0000","last_clean":"2024-03-16T09:41:27.804957+0000","last_became_active":"2024-03-16T09:39:56.178849+0000","last_became_peered":"2024-03-16T09:39:56.178849+0000","last_unstale":"2024-03-16T09:41:27.804957+0000","last_undegraded":"2024-03-16T09:41:27.804957+0000","last_fullsized":"2024-03-16T09:41:27.804957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:10.016099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:28.195243+0000","last_change":"2024-03-16T09:39:55.180859+0000","last_active":"2024-03-16T09:41:28.195243+0000","last_peered":"2024-03-16T09:41:28.195243+0000","last_clean":"2024-03-16T09:41:28.195243+0000","last_became_active":"2024-03-16T09:39:55.180514+0000","last_became_peered":"2024-03-16T09:39:55.180514+0000","last_unstale":"2024-03-16T09:41:28.195243+0000","last_undegraded":"2024-03-16T09:41:28.195243+0000","last_fullsized":"2024-03-16T09:41:28.195243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:46.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806604+0000","last_change":"2024-03-16T09:39:56.187626+0000","last_active":"2024-03-16T09:41:27.806604+0000","last_peered":"2024-03-16T09:41:27.806604+0000","last_clean":"2024-03-16T09:41:27.806604+0000","last_became_active":"2024-03-16T09:39:56.187284+0000","last_became_peered":"2024-03-16T09:39:56.187284+0000","last_unstale":"2024-03-16T09:41:27.806604+0000","last_undegraded":"2024-03-16T09:41:27.806604+0000","last_fullsized":"2024-03-16T09:41:27.806604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:23.511603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864099+0000","last_change":"2024-03-16T09:39:55.172912+0000","last_active":"2024-03-16T09:40:58.864099+0000","last_peered":"2024-03-16T09:40:58.864099+0000","last_clean":"2024-03-16T09:40:58.864099+0000","last_became_active":"2024-03-16T09:39:55.172592+0000","last_became_peered":"2024-03-16T09:39:55.172592+0000","last_unstale":"2024-03-16T09:40:58.864099+0000","last_undegraded":"2024-03-16T09:40:58.864099+0000","last_fullsized":"2024-03-16T09:40:58.864099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:24.344588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811086+0000","last_change":"2024-03-16T09:39:56.197652+0000","last_active":"2024-03-16T09:41:27.811086+0000","last_peered":"2024-03-16T09:41:27.811086+0000","last_clean":"2024-03-16T09:41:27.811086+0000","last_became_active":"2024-03-16T09:39:56.197230+0000","last_became_peered":"2024-03-16T09:39:56.197230+0000","last_unstale":"2024-03-16T09:41:27.811086+0000","last_undegraded":"2024-03-16T09:41:27.811086+0000","last_fullsized":"2024-03-16T09:41:27.811086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:45:08.661372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.869780+0000","last_change":"2024-03-16T09:40:58.869780+0000","last_active":"2024-03-16T09:40:58.869780+0000","last_peered":"2024-03-16T09:40:58.869780+0000","last_clean":"2024-03-16T09:40:58.869780+0000","last_became_active":"2024-03-16T09:40:58.866452+0000","last_became_peered":"2024-03-16T09:40:58.866452+0000","last_unstale":"2024-03-16T09:40:58.869780+0000","last_undegraded":"2024-03-16T09:40:58.869780+0000","last_fullsized":"2024-03-16T09:40:58.869780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:07:21.751383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.771232+0000","last_change":"2024-03-16T09:39:56.179326+0000","last_active":"2024-03-16T09:41:27.771232+0000","last_peered":"2024-03-16T09:41:27.771232+0000","last_clean":"2024-03-16T09:41:27.771232+0000","last_became_active":"2024-03-16T09:39:56.178681+0000","last_became_peered":"2024-03-16T09:39:56.178681+0000","last_unstale":"2024-03-16T09:41:27.771232+0000","last_undegraded":"2024-03-16T09:41:27.771232+0000","last_fullsized":"2024-03-16T09:41:27.771232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:26.137807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871338+0000","last_change":"2024-03-16T09:39:55.173820+0000","last_active":"2024-03-16T09:40:58.871338+0000","last_peered":"2024-03-16T09:40:58.871338+0000","last_clean":"2024-03-16T09:40:58.871338+0000","last_became_active":"2024-03-16T09:39:55.173472+0000","last_became_peered":"2024-03-16T09:39:55.173472+0000","last_unstale":"2024-03-16T09:40:58.871338+0000","last_undegraded":"2024-03-16T09:40:58.871338+0000","last_fullsized":"2024-03-16T09:40:58.871338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:14.082367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802385+0000","last_change":"2024-03-16T09:39:56.178535+0000","last_active":"2024-03-16T09:41:27.802385+0000","last_peered":"2024-03-16T09:41:27.802385+0000","last_clean":"2024-03-16T09:41:27.802385+0000","last_became_active":"2024-03-16T09:39:56.178210+0000","last_became_peered":"2024-03-16T09:39:56.178210+0000","last_unstale":"2024-03-16T09:41:27.802385+0000","last_undegraded":"2024-03-16T09:41:27.802385+0000","last_fullsized":"2024-03-16T09:41:27.802385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:18.680469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864978+0000","last_change":"2024-03-16T09:39:55.183074+0000","last_active":"2024-03-16T09:40:58.864978+0000","last_peered":"2024-03-16T09:40:58.864978+0000","last_clean":"2024-03-16T09:40:58.864978+0000","last_became_active":"2024-03-16T09:39:55.182695+0000","last_became_peered":"2024-03-16T09:39:55.182695+0000","last_unstale":"2024-03-16T09:40:58.864978+0000","last_undegraded":"2024-03-16T09:40:58.864978+0000","last_fullsized":"2024-03-16T09:40:58.864978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:51.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804379+0000","last_change":"2024-03-16T09:39:56.193434+0000","last_active":"2024-03-16T09:41:27.804379+0000","last_peered":"2024-03-16T09:41:27.804379+0000","last_clean":"2024-03-16T09:41:27.804379+0000","last_became_active":"2024-03-16T09:39:56.193051+0000","last_became_peered":"2024-03-16T09:39:56.193051+0000","last_unstale":"2024-03-16T09:41:27.804379+0000","last_undegraded":"2024-03-16T09:41:27.804379+0000","last_fullsized":"2024-03-16T09:41:27.804379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:12:26.767117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863710+0000","last_change":"2024-03-16T09:39:55.179916+0000","last_active":"2024-03-16T09:40:58.863710+0000","last_peered":"2024-03-16T09:40:58.863710+0000","last_clean":"2024-03-16T09:40:58.863710+0000","last_became_active":"2024-03-16T09:39:55.179635+0000","last_became_peered":"2024-03-16T09:39:55.179635+0000","last_unstale":"2024-03-16T09:40:58.863710+0000","last_undegraded":"2024-03-16T09:40:58.863710+0000","last_fullsized":"2024-03-16T09:40:58.863710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:08.955039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814067+0000","last_change":"2024-03-16T09:39:56.194044+0000","last_active":"2024-03-16T09:41:27.814067+0000","last_peered":"2024-03-16T09:41:27.814067+0000","last_clean":"2024-03-16T09:41:27.814067+0000","last_became_active":"2024-03-16T09:39:56.193490+0000","last_became_peered":"2024-03-16T09:39:56.193490+0000","last_unstale":"2024-03-16T09:41:27.814067+0000","last_undegraded":"2024-03-16T09:41:27.814067+0000","last_fullsized":"2024-03-16T09:41:27.814067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:19:59.916892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865495+0000","last_change":"2024-03-16T09:39:55.182652+0000","last_active":"2024-03-16T09:40:58.865495+0000","last_peered":"2024-03-16T09:40:58.865495+0000","last_clean":"2024-03-16T09:40:58.865495+0000","last_became_active":"2024-03-16T09:39:55.181432+0000","last_became_peered":"2024-03-16T09:39:55.181432+0000","last_unstale":"2024-03-16T09:40:58.865495+0000","last_undegraded":"2024-03-16T09:40:58.865495+0000","last_fullsized":"2024-03-16T09:40:58.865495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:34.959845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817361+0000","last_change":"2024-03-16T09:39:56.188970+0000","last_active":"2024-03-16T09:41:27.817361+0000","last_peered":"2024-03-16T09:41:27.817361+0000","last_clean":"2024-03-16T09:41:27.817361+0000","last_became_active":"2024-03-16T09:39:56.188607+0000","last_became_peered":"2024-03-16T09:39:56.188607+0000","last_unstale":"2024-03-16T09:41:27.817361+0000","last_undegraded":"2024-03-16T09:41:27.817361+0000","last_fullsized":"2024-03-16T09:41:27.817361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:39:24.532114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863497+0000","last_change":"2024-03-16T09:39:55.181908+0000","last_active":"2024-03-16T09:40:58.863497+0000","last_peered":"2024-03-16T09:40:58.863497+0000","last_clean":"2024-03-16T09:40:58.863497+0000","last_became_active":"2024-03-16T09:39:55.181055+0000","last_became_peered":"2024-03-16T09:39:55.181055+0000","last_unstale":"2024-03-16T09:40:58.863497+0000","last_undegraded":"2024-03-16T09:40:58.863497+0000","last_fullsized":"2024-03-16T09:40:58.863497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:38.256664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803448+0000","last_change":"2024-03-16T09:39:56.189743+0000","last_active":"2024-03-16T09:41:27.803448+0000","last_peered":"2024-03-16T09:41:27.803448+0000","last_clean":"2024-03-16T09:41:27.803448+0000","last_became_active":"2024-03-16T09:39:56.189339+0000","last_became_peered":"2024-03-16T09:39:56.189339+0000","last_unstale":"2024-03-16T09:41:27.803448+0000","last_undegraded":"2024-03-16T09:41:27.803448+0000","last_fullsized":"2024-03-16T09:41:27.803448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:54.694191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.874651+0000","last_change":"2024-03-16T09:40:58.874651+0000","last_active":"2024-03-16T09:40:58.874651+0000","last_peered":"2024-03-16T09:40:58.874651+0000","last_clean":"2024-03-16T09:40:58.874651+0000","last_became_active":"2024-03-16T09:40:58.872329+0000","last_became_peered":"2024-03-16T09:40:58.872329+0000","last_unstale":"2024-03-16T09:40:58.874651+0000","last_undegraded":"2024-03-16T09:40:58.874651+0000","last_fullsized":"2024-03-16T09:40:58.874651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:05.424197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808467+0000","last_change":"2024-03-16T09:39:56.181636+0000","last_active":"2024-03-16T09:41:27.808467+0000","last_peered":"2024-03-16T09:41:27.808467+0000","last_clean":"2024-03-16T09:41:27.808467+0000","last_became_active":"2024-03-16T09:39:56.181309+0000","last_became_peered":"2024-03-16T09:39:56.181309+0000","last_unstale":"2024-03-16T09:41:27.808467+0000","last_undegraded":"2024-03-16T09:41:27.808467+0000","last_fullsized":"2024-03-16T09:41:27.808467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:36.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802234+0000","last_change":"2024-03-16T09:39:56.186640+0000","last_active":"2024-03-16T09:41:27.802234+0000","last_peered":"2024-03-16T09:41:27.802234+0000","last_clean":"2024-03-16T09:41:27.802234+0000","last_became_active":"2024-03-16T09:39:56.186311+0000","last_became_peered":"2024-03-16T09:39:56.186311+0000","last_unstale":"2024-03-16T09:41:27.802234+0000","last_undegraded":"2024-03-16T09:41:27.802234+0000","last_fullsized":"2024-03-16T09:41:27.802234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:58:44.451068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803937+0000","last_change":"2024-03-16T09:39:56.196887+0000","last_active":"2024-03-16T09:41:27.803937+0000","last_peered":"2024-03-16T09:41:27.803937+0000","last_clean":"2024-03-16T09:41:27.803937+0000","last_became_active":"2024-03-16T09:39:56.196568+0000","last_became_peered":"2024-03-16T09:39:56.196568+0000","last_unstale":"2024-03-16T09:41:27.803937+0000","last_undegraded":"2024-03-16T09:41:27.803937+0000","last_fullsized":"2024-03-16T09:41:27.803937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:08:35.577630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804119+0000","last_change":"2024-03-16T09:39:56.186394+0000","last_active":"2024-03-16T09:41:27.804119+0000","last_peered":"2024-03-16T09:41:27.804119+0000","last_clean":"2024-03-16T09:41:27.804119+0000","last_became_active":"2024-03-16T09:39:56.186053+0000","last_became_peered":"2024-03-16T09:39:56.186053+0000","last_unstale":"2024-03-16T09:41:27.804119+0000","last_undegraded":"2024-03-16T09:41:27.804119+0000","last_fullsized":"2024-03-16T09:41:27.804119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:41.529601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811065+0000","last_change":"2024-03-16T09:39:56.197622+0000","last_active":"2024-03-16T09:41:27.811065+0000","last_peered":"2024-03-16T09:41:27.811065+0000","last_clean":"2024-03-16T09:41:27.811065+0000","last_became_active":"2024-03-16T09:39:56.196885+0000","last_became_peered":"2024-03-16T09:39:56.196885+0000","last_unstale":"2024-03-16T09:41:27.811065+0000","last_undegraded":"2024-03-16T09:41:27.811065+0000","last_fullsized":"2024-03-16T09:41:27.811065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:08:50.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811555+0000","last_change":"2024-03-16T09:40:58.868417+0000","last_active":"2024-03-16T09:41:27.811555+0000","last_peered":"2024-03-16T09:41:27.811555+0000","last_clean":"2024-03-16T09:41:27.811555+0000","last_became_active":"2024-03-16T09:40:58.868100+0000","last_became_peered":"2024-03-16T09:40:58.868100+0000","last_unstale":"2024-03-16T09:41:27.811555+0000","last_undegraded":"2024-03-16T09:41:27.811555+0000","last_fullsized":"2024-03-16T09:41:27.811555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:51.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.799783+0000","last_change":"2024-03-16T09:39:56.181262+0000","last_active":"2024-03-16T09:41:27.799783+0000","last_peered":"2024-03-16T09:41:27.799783+0000","last_clean":"2024-03-16T09:41:27.799783+0000","last_became_active":"2024-03-16T09:39:56.180916+0000","last_became_peered":"2024-03-16T09:39:56.180916+0000","last_unstale":"2024-03-16T09:41:27.799783+0000","last_undegraded":"2024-03-16T09:41:27.799783+0000","last_fullsized":"2024-03-16T09:41:27.799783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:34:45.056025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818672+0000","last_change":"2024-03-16T09:39:56.176621+0000","last_active":"2024-03-16T09:41:27.818672+0000","last_peered":"2024-03-16T09:41:27.818672+0000","last_clean":"2024-03-16T09:41:27.818672+0000","last_became_active":"2024-03-16T09:39:56.176296+0000","last_became_peered":"2024-03-16T09:39:56.176296+0000","last_unstale":"2024-03-16T09:41:27.818672+0000","last_undegraded":"2024-03-16T09:41:27.818672+0000","last_fullsized":"2024-03-16T09:41:27.818672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:54.673937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806225+0000","last_change":"2024-03-16T09:39:56.197689+0000","last_active":"2024-03-16T09:41:27.806225+0000","last_peered":"2024-03-16T09:41:27.806225+0000","last_clean":"2024-03-16T09:41:27.806225+0000","last_became_active":"2024-03-16T09:39:56.197343+0000","last_became_peered":"2024-03-16T09:39:56.197343+0000","last_unstale":"2024-03-16T09:41:27.806225+0000","last_undegraded":"2024-03-16T09:41:27.806225+0000","last_fullsized":"2024-03-16T09:41:27.806225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:28:10.218048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.799490+0000","last_change":"2024-03-16T09:39:56.193551+0000","last_active":"2024-03-16T09:41:27.799490+0000","last_peered":"2024-03-16T09:41:27.799490+0000","last_clean":"2024-03-16T09:41:27.799490+0000","last_became_active":"2024-03-16T09:39:56.192901+0000","last_became_peered":"2024-03-16T09:39:56.192901+0000","last_unstale":"2024-03-16T09:41:27.799490+0000","last_undegraded":"2024-03-16T09:41:27.799490+0000","last_fullsized":"2024-03-16T09:41:27.799490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:50:15.528370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808698+0000","last_change":"2024-03-16T09:39:56.188484+0000","last_active":"2024-03-16T09:41:27.808698+0000","last_peered":"2024-03-16T09:41:27.808698+0000","last_clean":"2024-03-16T09:41:27.808698+0000","last_became_active":"2024-03-16T09:39:56.188144+0000","last_became_peered":"2024-03-16T09:39:56.188144+0000","last_unstale":"2024-03-16T09:41:27.808698+0000","last_undegraded":"2024-03-16T09:41:27.808698+0000","last_fullsized":"2024-03-16T09:41:27.808698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:01:10.042765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818669+0000","last_change":"2024-03-16T09:39:56.187448+0000","last_active":"2024-03-16T09:41:27.818669+0000","last_peered":"2024-03-16T09:41:27.818669+0000","last_clean":"2024-03-16T09:41:27.818669+0000","last_became_active":"2024-03-16T09:39:56.186772+0000","last_became_peered":"2024-03-16T09:39:56.186772+0000","last_unstale":"2024-03-16T09:41:27.818669+0000","last_undegraded":"2024-03-16T09:41:27.818669+0000","last_fullsized":"2024-03-16T09:41:27.818669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:45:43.315011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814777+0000","last_change":"2024-03-16T09:39:56.191004+0000","last_active":"2024-03-16T09:41:27.814777+0000","last_peered":"2024-03-16T09:41:27.814777+0000","last_clean":"2024-03-16T09:41:27.814777+0000","last_became_active":"2024-03-16T09:39:56.190646+0000","last_became_peered":"2024-03-16T09:39:56.190646+0000","last_unstale":"2024-03-16T09:41:27.814777+0000","last_undegraded":"2024-03-16T09:41:27.814777+0000","last_fullsized":"2024-03-16T09:41:27.814777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:03.191129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803969+0000","last_change":"2024-03-16T09:39:56.197552+0000","last_active":"2024-03-16T09:41:27.803969+0000","last_peered":"2024-03-16T09:41:27.803969+0000","last_clean":"2024-03-16T09:41:27.803969+0000","last_became_active":"2024-03-16T09:39:56.197215+0000","last_became_peered":"2024-03-16T09:39:56.197215+0000","last_unstale":"2024-03-16T09:41:27.803969+0000","last_undegraded":"2024-03-16T09:41:27.803969+0000","last_fullsized":"2024-03-16T09:41:27.803969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:45.795328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64868,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":690,"num_read_kb":416,"num_write":169,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":460713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1113430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037812,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":460825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1053372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":460316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":520874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":461587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":522717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":461246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36876,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708276,"statfs":{"total":95995035648,"available":95957274624,"internally_reserved":0,"allocated":1576960,"data_stored":1052070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T09:41:52.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T09:41:53.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 -- 172.21.15.77:0/239126576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 msgr2=0x7f352c1044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 --2- 172.21.15.77:0/239126576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c1044a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f351c009900 tx=0x7f351c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:53.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 -- 172.21.15.77:0/239126576 shutdown_connections 2024-03-16T09:41:53.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 --2- 172.21.15.77:0/239126576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352c107310 0x7f352c109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 --2- 172.21.15.77:0/239126576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f352c1049e0 0x7f352c106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 --2- 172.21.15.77:0/239126576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c1044a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 -- 172.21.15.77:0/239126576 >> 172.21.15.77:0/239126576 conn(0x7f352c0fbac0 msgr2=0x7f352c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 -- 172.21.15.77:0/239126576 shutdown_connections 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.114+0000 7f353099d640 1 -- 172.21.15.77:0/239126576 wait complete. 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 Processor -- start 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 -- start start 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f352c1049e0 0x7f352c197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352c107310 0x7f352c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f352c10d150 con 0x7f352c1020b0 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f352c10cfd0 con 0x7f352c1049e0 2024-03-16T09:41:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f353099d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f352c10d2d0 con 0x7f352c107310 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37054/0 (socket says 172.21.15.77:37054) 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 -- 172.21.15.77:0/3609932720 learned_addr learned my addr 172.21.15.77:0/3609932720 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352affd640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f352c1049e0 0x7f352c197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352bfff640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352c107310 0x7f352c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 -- 172.21.15.77:0/3609932720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352c107310 msgr2=0x7f352c19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352c107310 0x7f352c19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 -- 172.21.15.77:0/3609932720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f352c1049e0 msgr2=0x7f352c197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f352c1049e0 0x7f352c197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.115+0000 7f352b7fe640 1 -- 172.21.15.77:0/3609932720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352c19f480 con 0x7f352c1020b0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.116+0000 7f352b7fe640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c197860 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f351c006010 tx=0x7f351c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.116+0000 7f3528ff9640 1 -- 172.21.15.77:0/3609932720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c043020 con 0x7f352c1020b0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.116+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352c19f710 con 0x7f352c1020b0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.116+0000 7f3528ff9640 1 -- 172.21.15.77:0/3609932720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f351c0043d0 con 0x7f352c1020b0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.116+0000 7f3528ff9640 1 -- 172.21.15.77:0/3609932720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c003a80 con 0x7f352c1020b0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.116+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f352c19fcb0 con 0x7f352c1020b0 2024-03-16T09:41:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.117+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f352c0fd5b0 con 0x7f352c1020b0 2024-03-16T09:41:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.118+0000 7f3528ff9640 1 -- 172.21.15.77:0/3609932720 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f351c046020 con 0x7f352c1020b0 2024-03-16T09:41:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.118+0000 7f3528ff9640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f35140a2800 0x7f35140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.118+0000 7f352affd640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f35140a2800 0x7f35140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.118+0000 7f3528ff9640 1 -- 172.21.15.77:0/3609932720 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f351c0f1230 con 0x7f352c1020b0 2024-03-16T09:41:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.119+0000 7f352affd640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f35140a2800 0x7f35140a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f352c198d70 tx=0x7f3520005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.122+0000 7f3528ff9640 1 -- 172.21.15.77:0/3609932720 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f351c04d020 con 0x7f352c1020b0 2024-03-16T09:41:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:52 smithi119 ceph-mon[35632]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:41:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:52 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3354665720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:52 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1682060792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:52 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/943994127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:53.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.293+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f352c061c20 con 0x7f35140a2800 2024-03-16T09:41:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:52 smithi129 ceph-mon[35841]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:41:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:52 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3354665720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:52 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1682060792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:53.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:52 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/943994127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:53.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.300+0000 7f3528ff9640 1 -- 172.21.15.77:0/3609932720 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258804 (secure 0 0 0) 0x7f352c061c20 con 0x7f35140a2800 2024-03-16T09:41:53.306 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:53.307 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T09:41:53.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f35140a2800 msgr2=0x7f35140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f35140a2800 0x7f35140a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f352c198d70 tx=0x7f3520005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 msgr2=0x7f352c197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c197860 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f351c006010 tx=0x7f351c005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 shutdown_connections 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f35140a2800 0x7f35140a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f352c107310 0x7f352c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f352c1049e0 0x7f352c197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 --2- 172.21.15.77:0/3609932720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c1020b0 0x7f352c197860 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 >> 172.21.15.77:0/3609932720 conn(0x7f352c0fbac0 msgr2=0x7f352c102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 shutdown_connections 2024-03-16T09:41:53.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.303+0000 7f353099d640 1 -- 172.21.15.77:0/3609932720 wait complete. 2024-03-16T09:41:53.317 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-16T09:41:51.800371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655470,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":816,"num_read_kb":525,"num_write":352,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358300,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583524,"statfs":{"total":1151940427776,"available":1151573528576,"internally_reserved":0,"allocated":13627392,"data_stored":7488704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":986,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003299"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.812919+0000","last_change":"2024-03-16T09:39:56.184389+0000","last_active":"2024-03-16T09:41:27.812919+0000","last_peered":"2024-03-16T09:41:27.812919+0000","last_clean":"2024-03-16T09:41:27.812919+0000","last_became_active":"2024-03-16T09:39:56.184129+0000","last_became_peered":"2024-03-16T09:39:56.184129+0000","last_unstale":"2024-03-16T09:41:27.812919+0000","last_undegraded":"2024-03-16T09:41:27.812919+0000","last_fullsized":"2024-03-16T09:41:27.812919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:33.786788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.800995+0000","last_change":"2024-03-16T09:39:56.193451+0000","last_active":"2024-03-16T09:41:27.800995+0000","last_peered":"2024-03-16T09:41:27.800995+0000","last_clean":"2024-03-16T09:41:27.800995+0000","last_became_active":"2024-03-16T09:39:56.187381+0000","last_became_peered":"2024-03-16T09:39:56.187381+0000","last_unstale":"2024-03-16T09:41:27.800995+0000","last_undegraded":"2024-03-16T09:41:27.800995+0000","last_fullsized":"2024-03-16T09:41:27.800995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:51.915654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816523+0000","last_change":"2024-03-16T09:39:56.183785+0000","last_active":"2024-03-16T09:41:27.816523+0000","last_peered":"2024-03-16T09:41:27.816523+0000","last_clean":"2024-03-16T09:41:27.816523+0000","last_became_active":"2024-03-16T09:39:56.183421+0000","last_became_peered":"2024-03-16T09:39:56.183421+0000","last_unstale":"2024-03-16T09:41:27.816523+0000","last_undegraded":"2024-03-16T09:41:27.816523+0000","last_fullsized":"2024-03-16T09:41:27.816523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:47.259139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806416+0000","last_change":"2024-03-16T09:39:56.195743+0000","last_active":"2024-03-16T09:41:27.806416+0000","last_peered":"2024-03-16T09:41:27.806416+0000","last_clean":"2024-03-16T09:41:27.806416+0000","last_became_active":"2024-03-16T09:39:56.194818+0000","last_became_peered":"2024-03-16T09:39:56.194818+0000","last_unstale":"2024-03-16T09:41:27.806416+0000","last_undegraded":"2024-03-16T09:41:27.806416+0000","last_fullsized":"2024-03-16T09:41:27.806416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:39:43.123879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.809286+0000","last_change":"2024-03-16T09:39:56.184970+0000","last_active":"2024-03-16T09:41:27.809286+0000","last_peered":"2024-03-16T09:41:27.809286+0000","last_clean":"2024-03-16T09:41:27.809286+0000","last_became_active":"2024-03-16T09:39:56.184632+0000","last_became_peered":"2024-03-16T09:39:56.184632+0000","last_unstale":"2024-03-16T09:41:27.809286+0000","last_undegraded":"2024-03-16T09:41:27.809286+0000","last_fullsized":"2024-03-16T09:41:27.809286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:46:51.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804440+0000","last_change":"2024-03-16T09:39:56.199688+0000","last_active":"2024-03-16T09:41:27.804440+0000","last_peered":"2024-03-16T09:41:27.804440+0000","last_clean":"2024-03-16T09:41:27.804440+0000","last_became_active":"2024-03-16T09:39:56.199094+0000","last_became_peered":"2024-03-16T09:39:56.199094+0000","last_unstale":"2024-03-16T09:41:27.804440+0000","last_undegraded":"2024-03-16T09:41:27.804440+0000","last_fullsized":"2024-03-16T09:41:27.804440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:10.855504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.809387+0000","last_change":"2024-03-16T09:39:56.180816+0000","last_active":"2024-03-16T09:41:27.809387+0000","last_peered":"2024-03-16T09:41:27.809387+0000","last_clean":"2024-03-16T09:41:27.809387+0000","last_became_active":"2024-03-16T09:39:56.180499+0000","last_became_peered":"2024-03-16T09:39:56.180499+0000","last_unstale":"2024-03-16T09:41:27.809387+0000","last_undegraded":"2024-03-16T09:41:27.809387+0000","last_fullsized":"2024-03-16T09:41:27.809387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:32:09.444494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802950+0000","last_change":"2024-03-16T09:39:56.189960+0000","last_active":"2024-03-16T09:41:27.802950+0000","last_peered":"2024-03-16T09:41:27.802950+0000","last_clean":"2024-03-16T09:41:27.802950+0000","last_became_active":"2024-03-16T09:39:56.189609+0000","last_became_peered":"2024-03-16T09:39:56.189609+0000","last_unstale":"2024-03-16T09:41:27.802950+0000","last_undegraded":"2024-03-16T09:41:27.802950+0000","last_fullsized":"2024-03-16T09:41:27.802950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:50.219703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816982+0000","last_change":"2024-03-16T09:40:58.867996+0000","last_active":"2024-03-16T09:41:27.816982+0000","last_peered":"2024-03-16T09:41:27.816982+0000","last_clean":"2024-03-16T09:41:27.816982+0000","last_became_active":"2024-03-16T09:40:58.867619+0000","last_became_peered":"2024-03-16T09:40:58.867619+0000","last_unstale":"2024-03-16T09:41:27.816982+0000","last_undegraded":"2024-03-16T09:41:27.816982+0000","last_fullsized":"2024-03-16T09:41:27.816982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:36.972983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804258+0000","last_change":"2024-03-16T09:39:56.201584+0000","last_active":"2024-03-16T09:41:27.804258+0000","last_peered":"2024-03-16T09:41:27.804258+0000","last_clean":"2024-03-16T09:41:27.804258+0000","last_became_active":"2024-03-16T09:39:56.201320+0000","last_became_peered":"2024-03-16T09:39:56.201320+0000","last_unstale":"2024-03-16T09:41:27.804258+0000","last_undegraded":"2024-03-16T09:41:27.804258+0000","last_fullsized":"2024-03-16T09:41:27.804258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:18:00.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817302+0000","last_change":"2024-03-16T09:39:56.191125+0000","last_active":"2024-03-16T09:41:27.817302+0000","last_peered":"2024-03-16T09:41:27.817302+0000","last_clean":"2024-03-16T09:41:27.817302+0000","last_became_active":"2024-03-16T09:39:56.190795+0000","last_became_peered":"2024-03-16T09:39:56.190795+0000","last_unstale":"2024-03-16T09:41:27.817302+0000","last_undegraded":"2024-03-16T09:41:27.817302+0000","last_fullsized":"2024-03-16T09:41:27.817302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:44:10.894995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805793+0000","last_change":"2024-03-16T09:39:56.195652+0000","last_active":"2024-03-16T09:41:27.805793+0000","last_peered":"2024-03-16T09:41:27.805793+0000","last_clean":"2024-03-16T09:41:27.805793+0000","last_became_active":"2024-03-16T09:39:56.195270+0000","last_became_peered":"2024-03-16T09:39:56.195270+0000","last_unstale":"2024-03-16T09:41:27.805793+0000","last_undegraded":"2024-03-16T09:41:27.805793+0000","last_fullsized":"2024-03-16T09:41:27.805793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:03:41.546991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810598+0000","last_change":"2024-03-16T09:39:56.184622+0000","last_active":"2024-03-16T09:41:27.810598+0000","last_peered":"2024-03-16T09:41:27.810598+0000","last_clean":"2024-03-16T09:41:27.810598+0000","last_became_active":"2024-03-16T09:39:56.183532+0000","last_became_peered":"2024-03-16T09:39:56.183532+0000","last_unstale":"2024-03-16T09:41:27.810598+0000","last_undegraded":"2024-03-16T09:41:27.810598+0000","last_fullsized":"2024-03-16T09:41:27.810598+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:11:24.574088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808922+0000","last_change":"2024-03-16T09:39:56.184393+0000","last_active":"2024-03-16T09:41:27.808922+0000","last_peered":"2024-03-16T09:41:27.808922+0000","last_clean":"2024-03-16T09:41:27.808922+0000","last_became_active":"2024-03-16T09:39:56.183783+0000","last_became_peered":"2024-03-16T09:39:56.183783+0000","last_unstale":"2024-03-16T09:41:27.808922+0000","last_undegraded":"2024-03-16T09:41:27.808922+0000","last_fullsized":"2024-03-16T09:41:27.808922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:16:22.131116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811183+0000","last_change":"2024-03-16T09:39:56.197365+0000","last_active":"2024-03-16T09:41:27.811183+0000","last_peered":"2024-03-16T09:41:27.811183+0000","last_clean":"2024-03-16T09:41:27.811183+0000","last_became_active":"2024-03-16T09:39:56.197020+0000","last_became_peered":"2024-03-16T09:39:56.197020+0000","last_unstale":"2024-03-16T09:41:27.811183+0000","last_undegraded":"2024-03-16T09:41:27.811183+0000","last_fullsized":"2024-03-16T09:41:27.811183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:29.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817996+0000","last_change":"2024-03-16T09:39:56.185513+0000","last_active":"2024-03-16T09:41:27.817996+0000","last_peered":"2024-03-16T09:41:27.817996+0000","last_clean":"2024-03-16T09:41:27.817996+0000","last_became_active":"2024-03-16T09:39:56.184384+0000","last_became_peered":"2024-03-16T09:39:56.184384+0000","last_unstale":"2024-03-16T09:41:27.817996+0000","last_undegraded":"2024-03-16T09:41:27.817996+0000","last_fullsized":"2024-03-16T09:41:27.817996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:29.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.813059+0000","last_change":"2024-03-16T09:39:56.184536+0000","last_active":"2024-03-16T09:41:27.813059+0000","last_peered":"2024-03-16T09:41:27.813059+0000","last_clean":"2024-03-16T09:41:27.813059+0000","last_became_active":"2024-03-16T09:39:56.184208+0000","last_became_peered":"2024-03-16T09:39:56.184208+0000","last_unstale":"2024-03-16T09:41:27.813059+0000","last_undegraded":"2024-03-16T09:41:27.813059+0000","last_fullsized":"2024-03-16T09:41:27.813059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:14.025625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.796187+0000","last_change":"2024-03-16T09:39:56.184331+0000","last_active":"2024-03-16T09:41:27.796187+0000","last_peered":"2024-03-16T09:41:27.796187+0000","last_clean":"2024-03-16T09:41:27.796187+0000","last_became_active":"2024-03-16T09:39:56.183919+0000","last_became_peered":"2024-03-16T09:39:56.183919+0000","last_unstale":"2024-03-16T09:41:27.796187+0000","last_undegraded":"2024-03-16T09:41:27.796187+0000","last_fullsized":"2024-03-16T09:41:27.796187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:59.692854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802842+0000","last_change":"2024-03-16T09:39:56.185607+0000","last_active":"2024-03-16T09:41:27.802842+0000","last_peered":"2024-03-16T09:41:27.802842+0000","last_clean":"2024-03-16T09:41:27.802842+0000","last_became_active":"2024-03-16T09:39:56.185278+0000","last_became_peered":"2024-03-16T09:39:56.185278+0000","last_unstale":"2024-03-16T09:41:27.802842+0000","last_undegraded":"2024-03-16T09:41:27.802842+0000","last_fullsized":"2024-03-16T09:41:27.802842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:20.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860105+0000","last_change":"2024-03-16T09:39:55.180852+0000","last_active":"2024-03-16T09:40:58.860105+0000","last_peered":"2024-03-16T09:40:58.860105+0000","last_clean":"2024-03-16T09:40:58.860105+0000","last_became_active":"2024-03-16T09:39:55.180496+0000","last_became_peered":"2024-03-16T09:39:55.180496+0000","last_unstale":"2024-03-16T09:40:58.860105+0000","last_undegraded":"2024-03-16T09:40:58.860105+0000","last_fullsized":"2024-03-16T09:40:58.860105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:15:28.548458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805721+0000","last_change":"2024-03-16T09:39:56.190110+0000","last_active":"2024-03-16T09:41:27.805721+0000","last_peered":"2024-03-16T09:41:27.805721+0000","last_clean":"2024-03-16T09:41:27.805721+0000","last_became_active":"2024-03-16T09:39:56.189711+0000","last_became_peered":"2024-03-16T09:39:56.189711+0000","last_unstale":"2024-03-16T09:41:27.805721+0000","last_undegraded":"2024-03-16T09:41:27.805721+0000","last_fullsized":"2024-03-16T09:41:27.805721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:17.353457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871621+0000","last_change":"2024-03-16T09:39:55.171142+0000","last_active":"2024-03-16T09:40:58.871621+0000","last_peered":"2024-03-16T09:40:58.871621+0000","last_clean":"2024-03-16T09:40:58.871621+0000","last_became_active":"2024-03-16T09:39:55.170793+0000","last_became_peered":"2024-03-16T09:39:55.170793+0000","last_unstale":"2024-03-16T09:40:58.871621+0000","last_undegraded":"2024-03-16T09:40:58.871621+0000","last_fullsized":"2024-03-16T09:40:58.871621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:36.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816268+0000","last_change":"2024-03-16T09:39:56.186802+0000","last_active":"2024-03-16T09:41:27.816268+0000","last_peered":"2024-03-16T09:41:27.816268+0000","last_clean":"2024-03-16T09:41:27.816268+0000","last_became_active":"2024-03-16T09:39:56.186398+0000","last_became_peered":"2024-03-16T09:39:56.186398+0000","last_unstale":"2024-03-16T09:41:27.816268+0000","last_undegraded":"2024-03-16T09:41:27.816268+0000","last_fullsized":"2024-03-16T09:41:27.816268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:05:56.209864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864171+0000","last_change":"2024-03-16T09:39:55.181894+0000","last_active":"2024-03-16T09:40:58.864171+0000","last_peered":"2024-03-16T09:40:58.864171+0000","last_clean":"2024-03-16T09:40:58.864171+0000","last_became_active":"2024-03-16T09:39:55.179434+0000","last_became_peered":"2024-03-16T09:39:55.179434+0000","last_unstale":"2024-03-16T09:40:58.864171+0000","last_undegraded":"2024-03-16T09:40:58.864171+0000","last_fullsized":"2024-03-16T09:40:58.864171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:16.153928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818117+0000","last_change":"2024-03-16T09:39:56.188464+0000","last_active":"2024-03-16T09:41:27.818117+0000","last_peered":"2024-03-16T09:41:27.818117+0000","last_clean":"2024-03-16T09:41:27.818117+0000","last_became_active":"2024-03-16T09:39:56.188125+0000","last_became_peered":"2024-03-16T09:39:56.188125+0000","last_unstale":"2024-03-16T09:41:27.818117+0000","last_undegraded":"2024-03-16T09:41:27.818117+0000","last_fullsized":"2024-03-16T09:41:27.818117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:46:49.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863856+0000","last_change":"2024-03-16T09:39:55.179379+0000","last_active":"2024-03-16T09:40:58.863856+0000","last_peered":"2024-03-16T09:40:58.863856+0000","last_clean":"2024-03-16T09:40:58.863856+0000","last_became_active":"2024-03-16T09:39:55.178755+0000","last_became_peered":"2024-03-16T09:39:55.178755+0000","last_unstale":"2024-03-16T09:40:58.863856+0000","last_undegraded":"2024-03-16T09:40:58.863856+0000","last_fullsized":"2024-03-16T09:40:58.863856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:07.131417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805630+0000","last_change":"2024-03-16T09:39:56.196655+0000","last_active":"2024-03-16T09:41:27.805630+0000","last_peered":"2024-03-16T09:41:27.805630+0000","last_clean":"2024-03-16T09:41:27.805630+0000","last_became_active":"2024-03-16T09:39:56.196344+0000","last_became_peered":"2024-03-16T09:39:56.196344+0000","last_unstale":"2024-03-16T09:41:27.805630+0000","last_undegraded":"2024-03-16T09:41:27.805630+0000","last_fullsized":"2024-03-16T09:41:27.805630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:12:03.920319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":134,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:45.842807+0000","last_change":"2024-03-16T09:39:55.182012+0000","last_active":"2024-03-16T09:41:45.842807+0000","last_peered":"2024-03-16T09:41:45.842807+0000","last_clean":"2024-03-16T09:41:45.842807+0000","last_became_active":"2024-03-16T09:39:55.178651+0000","last_became_peered":"2024-03-16T09:39:55.178651+0000","last_unstale":"2024-03-16T09:41:45.842807+0000","last_undegraded":"2024-03-16T09:41:45.842807+0000","last_fullsized":"2024-03-16T09:41:45.842807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:50:19.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.795850+0000","last_change":"2024-03-16T09:39:56.179645+0000","last_active":"2024-03-16T09:41:27.795850+0000","last_peered":"2024-03-16T09:41:27.795850+0000","last_clean":"2024-03-16T09:41:27.795850+0000","last_became_active":"2024-03-16T09:39:56.179257+0000","last_became_peered":"2024-03-16T09:39:56.179257+0000","last_unstale":"2024-03-16T09:41:27.795850+0000","last_undegraded":"2024-03-16T09:41:27.795850+0000","last_fullsized":"2024-03-16T09:41:27.795850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:30:09.104630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.218444+0000","last_change":"2024-03-16T09:39:55.178257+0000","last_active":"2024-03-16T09:40:59.218444+0000","last_peered":"2024-03-16T09:40:59.218444+0000","last_clean":"2024-03-16T09:40:59.218444+0000","last_became_active":"2024-03-16T09:39:55.177984+0000","last_became_peered":"2024-03-16T09:39:55.177984+0000","last_unstale":"2024-03-16T09:40:59.218444+0000","last_undegraded":"2024-03-16T09:40:59.218444+0000","last_fullsized":"2024-03-16T09:40:59.218444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.385624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804510+0000","last_change":"2024-03-16T09:39:56.196821+0000","last_active":"2024-03-16T09:41:27.804510+0000","last_peered":"2024-03-16T09:41:27.804510+0000","last_clean":"2024-03-16T09:41:27.804510+0000","last_became_active":"2024-03-16T09:39:56.196522+0000","last_became_peered":"2024-03-16T09:39:56.196522+0000","last_unstale":"2024-03-16T09:41:27.804510+0000","last_undegraded":"2024-03-16T09:41:27.804510+0000","last_fullsized":"2024-03-16T09:41:27.804510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:31.103530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865998+0000","last_change":"2024-03-16T09:39:55.170290+0000","last_active":"2024-03-16T09:40:58.865998+0000","last_peered":"2024-03-16T09:40:58.865998+0000","last_clean":"2024-03-16T09:40:58.865998+0000","last_became_active":"2024-03-16T09:39:55.169956+0000","last_became_peered":"2024-03-16T09:39:55.169956+0000","last_unstale":"2024-03-16T09:40:58.865998+0000","last_undegraded":"2024-03-16T09:40:58.865998+0000","last_fullsized":"2024-03-16T09:40:58.865998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:01:31.606917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.801687+0000","last_change":"2024-03-16T09:39:56.189155+0000","last_active":"2024-03-16T09:41:27.801687+0000","last_peered":"2024-03-16T09:41:27.801687+0000","last_clean":"2024-03-16T09:41:27.801687+0000","last_became_active":"2024-03-16T09:39:56.188827+0000","last_became_peered":"2024-03-16T09:39:56.188827+0000","last_unstale":"2024-03-16T09:41:27.801687+0000","last_undegraded":"2024-03-16T09:41:27.801687+0000","last_fullsized":"2024-03-16T09:41:27.801687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:54.370754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863093+0000","last_change":"2024-03-16T09:39:55.181886+0000","last_active":"2024-03-16T09:40:58.863093+0000","last_peered":"2024-03-16T09:40:58.863093+0000","last_clean":"2024-03-16T09:40:58.863093+0000","last_became_active":"2024-03-16T09:39:55.179836+0000","last_became_peered":"2024-03-16T09:39:55.179836+0000","last_unstale":"2024-03-16T09:40:58.863093+0000","last_undegraded":"2024-03-16T09:40:58.863093+0000","last_fullsized":"2024-03-16T09:40:58.863093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:52:38.772861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.795474+0000","last_change":"2024-03-16T09:39:56.185892+0000","last_active":"2024-03-16T09:41:27.795474+0000","last_peered":"2024-03-16T09:41:27.795474+0000","last_clean":"2024-03-16T09:41:27.795474+0000","last_became_active":"2024-03-16T09:39:56.185459+0000","last_became_peered":"2024-03-16T09:39:56.185459+0000","last_unstale":"2024-03-16T09:41:27.795474+0000","last_undegraded":"2024-03-16T09:41:27.795474+0000","last_fullsized":"2024-03-16T09:41:27.795474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:10.421054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":134,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:48.845916+0000","last_change":"2024-03-16T09:39:55.177607+0000","last_active":"2024-03-16T09:41:48.845916+0000","last_peered":"2024-03-16T09:41:48.845916+0000","last_clean":"2024-03-16T09:41:48.845916+0000","last_became_active":"2024-03-16T09:39:55.177265+0000","last_became_peered":"2024-03-16T09:39:55.177265+0000","last_unstale":"2024-03-16T09:41:48.845916+0000","last_undegraded":"2024-03-16T09:41:48.845916+0000","last_fullsized":"2024-03-16T09:41:48.845916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:47.849041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818050+0000","last_change":"2024-03-16T09:39:56.189172+0000","last_active":"2024-03-16T09:41:27.818050+0000","last_peered":"2024-03-16T09:41:27.818050+0000","last_clean":"2024-03-16T09:41:27.818050+0000","last_became_active":"2024-03-16T09:39:56.188798+0000","last_became_peered":"2024-03-16T09:39:56.188798+0000","last_unstale":"2024-03-16T09:41:27.818050+0000","last_undegraded":"2024-03-16T09:41:27.818050+0000","last_fullsized":"2024-03-16T09:41:27.818050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:02:11.119293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860780+0000","last_change":"2024-03-16T09:39:55.180369+0000","last_active":"2024-03-16T09:40:58.860780+0000","last_peered":"2024-03-16T09:40:58.860780+0000","last_clean":"2024-03-16T09:40:58.860780+0000","last_became_active":"2024-03-16T09:39:55.180053+0000","last_became_peered":"2024-03-16T09:39:55.180053+0000","last_unstale":"2024-03-16T09:40:58.860780+0000","last_undegraded":"2024-03-16T09:40:58.860780+0000","last_fullsized":"2024-03-16T09:40:58.860780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:44.990028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811041+0000","last_change":"2024-03-16T09:39:56.198454+0000","last_active":"2024-03-16T09:41:27.811041+0000","last_peered":"2024-03-16T09:41:27.811041+0000","last_clean":"2024-03-16T09:41:27.811041+0000","last_became_active":"2024-03-16T09:39:56.198097+0000","last_became_peered":"2024-03-16T09:39:56.198097+0000","last_unstale":"2024-03-16T09:41:27.811041+0000","last_undegraded":"2024-03-16T09:41:27.811041+0000","last_fullsized":"2024-03-16T09:41:27.811041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:12.494363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863700+0000","last_change":"2024-03-16T09:39:55.174221+0000","last_active":"2024-03-16T09:40:58.863700+0000","last_peered":"2024-03-16T09:40:58.863700+0000","last_clean":"2024-03-16T09:40:58.863700+0000","last_became_active":"2024-03-16T09:39:55.173920+0000","last_became_peered":"2024-03-16T09:39:55.173920+0000","last_unstale":"2024-03-16T09:40:58.863700+0000","last_undegraded":"2024-03-16T09:40:58.863700+0000","last_fullsized":"2024-03-16T09:40:58.863700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:25.108414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816889+0000","last_change":"2024-03-16T09:39:56.196849+0000","last_active":"2024-03-16T09:41:27.816889+0000","last_peered":"2024-03-16T09:41:27.816889+0000","last_clean":"2024-03-16T09:41:27.816889+0000","last_became_active":"2024-03-16T09:39:56.196515+0000","last_became_peered":"2024-03-16T09:39:56.196515+0000","last_unstale":"2024-03-16T09:41:27.816889+0000","last_undegraded":"2024-03-16T09:41:27.816889+0000","last_fullsized":"2024-03-16T09:41:27.816889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:03.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":285,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:49.846725+0000","last_change":"2024-03-16T09:39:55.177788+0000","last_active":"2024-03-16T09:41:49.846725+0000","last_peered":"2024-03-16T09:41:49.846725+0000","last_clean":"2024-03-16T09:41:49.846725+0000","last_became_active":"2024-03-16T09:39:55.177443+0000","last_became_peered":"2024-03-16T09:39:55.177443+0000","last_unstale":"2024-03-16T09:41:49.846725+0000","last_undegraded":"2024-03-16T09:41:49.846725+0000","last_fullsized":"2024-03-16T09:41:49.846725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:39.221712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60765,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":232,"num_read_kb":175,"num_write":35,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.807296+0000","last_change":"2024-03-16T09:39:56.178976+0000","last_active":"2024-03-16T09:41:27.807296+0000","last_peered":"2024-03-16T09:41:27.807296+0000","last_clean":"2024-03-16T09:41:27.807296+0000","last_became_active":"2024-03-16T09:39:56.178455+0000","last_became_peered":"2024-03-16T09:39:56.178455+0000","last_unstale":"2024-03-16T09:41:27.807296+0000","last_undegraded":"2024-03-16T09:41:27.807296+0000","last_fullsized":"2024-03-16T09:41:27.807296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:52:08.064792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865191+0000","last_change":"2024-03-16T09:39:55.182579+0000","last_active":"2024-03-16T09:40:58.865191+0000","last_peered":"2024-03-16T09:40:58.865191+0000","last_clean":"2024-03-16T09:40:58.865191+0000","last_became_active":"2024-03-16T09:39:55.181298+0000","last_became_peered":"2024-03-16T09:39:55.181298+0000","last_unstale":"2024-03-16T09:40:58.865191+0000","last_undegraded":"2024-03-16T09:40:58.865191+0000","last_fullsized":"2024-03-16T09:40:58.865191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:35.768812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814563+0000","last_change":"2024-03-16T09:39:56.188519+0000","last_active":"2024-03-16T09:41:27.814563+0000","last_peered":"2024-03-16T09:41:27.814563+0000","last_clean":"2024-03-16T09:41:27.814563+0000","last_became_active":"2024-03-16T09:39:56.188031+0000","last_became_peered":"2024-03-16T09:39:56.188031+0000","last_unstale":"2024-03-16T09:41:27.814563+0000","last_undegraded":"2024-03-16T09:41:27.814563+0000","last_fullsized":"2024-03-16T09:41:27.814563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:20:31.765657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864304+0000","last_change":"2024-03-16T09:39:55.177523+0000","last_active":"2024-03-16T09:40:58.864304+0000","last_peered":"2024-03-16T09:40:58.864304+0000","last_clean":"2024-03-16T09:40:58.864304+0000","last_became_active":"2024-03-16T09:39:55.177143+0000","last_became_peered":"2024-03-16T09:39:55.177143+0000","last_unstale":"2024-03-16T09:40:58.864304+0000","last_undegraded":"2024-03-16T09:40:58.864304+0000","last_fullsized":"2024-03-16T09:40:58.864304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:05.357611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.813082+0000","last_change":"2024-03-16T09:40:58.867151+0000","last_active":"2024-03-16T09:41:27.813082+0000","last_peered":"2024-03-16T09:41:27.813082+0000","last_clean":"2024-03-16T09:41:27.813082+0000","last_became_active":"2024-03-16T09:40:58.866756+0000","last_became_peered":"2024-03-16T09:40:58.866756+0000","last_unstale":"2024-03-16T09:41:27.813082+0000","last_undegraded":"2024-03-16T09:41:27.813082+0000","last_fullsized":"2024-03-16T09:41:27.813082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:53.323964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:47.866162+0000","last_change":"2024-03-16T09:39:55.175419+0000","last_active":"2024-03-16T09:41:47.866162+0000","last_peered":"2024-03-16T09:41:47.866162+0000","last_clean":"2024-03-16T09:41:47.866162+0000","last_became_active":"2024-03-16T09:39:55.175126+0000","last_became_peered":"2024-03-16T09:39:55.175126+0000","last_unstale":"2024-03-16T09:41:47.866162+0000","last_undegraded":"2024-03-16T09:41:47.866162+0000","last_fullsized":"2024-03-16T09:41:47.866162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:11.696911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.812473+0000","last_change":"2024-03-16T09:39:56.176714+0000","last_active":"2024-03-16T09:41:27.812473+0000","last_peered":"2024-03-16T09:41:27.812473+0000","last_clean":"2024-03-16T09:41:27.812473+0000","last_became_active":"2024-03-16T09:39:56.176245+0000","last_became_peered":"2024-03-16T09:39:56.176245+0000","last_unstale":"2024-03-16T09:41:27.812473+0000","last_undegraded":"2024-03-16T09:41:27.812473+0000","last_fullsized":"2024-03-16T09:41:27.812473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:32.430207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863725+0000","last_change":"2024-03-16T09:39:55.181486+0000","last_active":"2024-03-16T09:40:58.863725+0000","last_peered":"2024-03-16T09:40:58.863725+0000","last_clean":"2024-03-16T09:40:58.863725+0000","last_became_active":"2024-03-16T09:39:55.180128+0000","last_became_peered":"2024-03-16T09:39:55.180128+0000","last_unstale":"2024-03-16T09:40:58.863725+0000","last_undegraded":"2024-03-16T09:40:58.863725+0000","last_fullsized":"2024-03-16T09:40:58.863725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:45.323870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803486+0000","last_change":"2024-03-16T09:39:56.196692+0000","last_active":"2024-03-16T09:41:27.803486+0000","last_peered":"2024-03-16T09:41:27.803486+0000","last_clean":"2024-03-16T09:41:27.803486+0000","last_became_active":"2024-03-16T09:39:56.196403+0000","last_became_peered":"2024-03-16T09:39:56.196403+0000","last_unstale":"2024-03-16T09:41:27.803486+0000","last_undegraded":"2024-03-16T09:41:27.803486+0000","last_fullsized":"2024-03-16T09:41:27.803486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:00:29.426476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871091+0000","last_change":"2024-03-16T09:40:58.870587+0000","last_active":"2024-03-16T09:40:58.871091+0000","last_peered":"2024-03-16T09:40:58.871091+0000","last_clean":"2024-03-16T09:40:58.871091+0000","last_became_active":"2024-03-16T09:40:58.864977+0000","last_became_peered":"2024-03-16T09:40:58.864977+0000","last_unstale":"2024-03-16T09:40:58.871091+0000","last_undegraded":"2024-03-16T09:40:58.871091+0000","last_fullsized":"2024-03-16T09:40:58.871091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:18.189307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.798760+0000","last_change":"2024-03-16T09:40:58.866074+0000","last_active":"2024-03-16T09:41:27.798760+0000","last_peered":"2024-03-16T09:41:27.798760+0000","last_clean":"2024-03-16T09:41:27.798760+0000","last_became_active":"2024-03-16T09:40:58.865711+0000","last_became_peered":"2024-03-16T09:40:58.865711+0000","last_unstale":"2024-03-16T09:41:27.798760+0000","last_undegraded":"2024-03-16T09:41:27.798760+0000","last_fullsized":"2024-03-16T09:41:27.798760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:57.137802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.859988+0000","last_change":"2024-03-16T09:39:55.175666+0000","last_active":"2024-03-16T09:40:58.859988+0000","last_peered":"2024-03-16T09:40:58.859988+0000","last_clean":"2024-03-16T09:40:58.859988+0000","last_became_active":"2024-03-16T09:39:55.175337+0000","last_became_peered":"2024-03-16T09:39:55.175337+0000","last_unstale":"2024-03-16T09:40:58.859988+0000","last_undegraded":"2024-03-16T09:40:58.859988+0000","last_fullsized":"2024-03-16T09:40:58.859988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:45:13.935669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810754+0000","last_change":"2024-03-16T09:39:56.184878+0000","last_active":"2024-03-16T09:41:27.810754+0000","last_peered":"2024-03-16T09:41:27.810754+0000","last_clean":"2024-03-16T09:41:27.810754+0000","last_became_active":"2024-03-16T09:39:56.184480+0000","last_became_peered":"2024-03-16T09:39:56.184480+0000","last_unstale":"2024-03-16T09:41:27.810754+0000","last_undegraded":"2024-03-16T09:41:27.810754+0000","last_fullsized":"2024-03-16T09:41:27.810754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:40:01.286993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860584+0000","last_change":"2024-03-16T09:39:55.179189+0000","last_active":"2024-03-16T09:40:58.860584+0000","last_peered":"2024-03-16T09:40:58.860584+0000","last_clean":"2024-03-16T09:40:58.860584+0000","last_became_active":"2024-03-16T09:39:55.178859+0000","last_became_peered":"2024-03-16T09:39:55.178859+0000","last_unstale":"2024-03-16T09:40:58.860584+0000","last_undegraded":"2024-03-16T09:40:58.860584+0000","last_fullsized":"2024-03-16T09:40:58.860584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:54:15.498386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.788321+0000","last_change":"2024-03-16T09:39:56.200665+0000","last_active":"2024-03-16T09:41:27.788321+0000","last_peered":"2024-03-16T09:41:27.788321+0000","last_clean":"2024-03-16T09:41:27.788321+0000","last_became_active":"2024-03-16T09:39:56.200326+0000","last_became_peered":"2024-03-16T09:39:56.200326+0000","last_unstale":"2024-03-16T09:41:27.788321+0000","last_undegraded":"2024-03-16T09:41:27.788321+0000","last_fullsized":"2024-03-16T09:41:27.788321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:34.778630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806467+0000","last_change":"2024-03-16T09:39:56.182670+0000","last_active":"2024-03-16T09:41:27.806467+0000","last_peered":"2024-03-16T09:41:27.806467+0000","last_clean":"2024-03-16T09:41:27.806467+0000","last_became_active":"2024-03-16T09:39:56.182009+0000","last_became_peered":"2024-03-16T09:39:56.182009+0000","last_unstale":"2024-03-16T09:41:27.806467+0000","last_undegraded":"2024-03-16T09:41:27.806467+0000","last_fullsized":"2024-03-16T09:41:27.806467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:28:03.106071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:49.869064+0000","last_change":"2024-03-16T09:39:55.171609+0000","last_active":"2024-03-16T09:41:49.869064+0000","last_peered":"2024-03-16T09:41:49.869064+0000","last_clean":"2024-03-16T09:41:49.869064+0000","last_became_active":"2024-03-16T09:39:55.171301+0000","last_became_peered":"2024-03-16T09:39:55.171301+0000","last_unstale":"2024-03-16T09:41:49.869064+0000","last_undegraded":"2024-03-16T09:41:49.869064+0000","last_fullsized":"2024-03-16T09:41:49.869064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:28.265342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.787638+0000","last_change":"2024-03-16T09:40:58.865311+0000","last_active":"2024-03-16T09:41:27.787638+0000","last_peered":"2024-03-16T09:41:27.787638+0000","last_clean":"2024-03-16T09:41:27.787638+0000","last_became_active":"2024-03-16T09:40:58.864918+0000","last_became_peered":"2024-03-16T09:40:58.864918+0000","last_unstale":"2024-03-16T09:41:27.787638+0000","last_undegraded":"2024-03-16T09:41:27.787638+0000","last_fullsized":"2024-03-16T09:41:27.787638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863236+0000","last_change":"2024-03-16T09:39:55.172824+0000","last_active":"2024-03-16T09:40:58.863236+0000","last_peered":"2024-03-16T09:40:58.863236+0000","last_clean":"2024-03-16T09:40:58.863236+0000","last_became_active":"2024-03-16T09:39:55.172493+0000","last_became_peered":"2024-03-16T09:39:55.172493+0000","last_unstale":"2024-03-16T09:40:58.863236+0000","last_undegraded":"2024-03-16T09:40:58.863236+0000","last_fullsized":"2024-03-16T09:40:58.863236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:39.188176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":346,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860504+0000","last_change":"2024-03-16T09:38:43.693644+0000","last_active":"2024-03-16T09:40:58.860504+0000","last_peered":"2024-03-16T09:40:58.860504+0000","last_clean":"2024-03-16T09:40:58.860504+0000","last_became_active":"2024-03-16T09:38:43.693202+0000","last_became_peered":"2024-03-16T09:38:43.693202+0000","last_unstale":"2024-03-16T09:40:58.860504+0000","last_undegraded":"2024-03-16T09:40:58.860504+0000","last_fullsized":"2024-03-16T09:40:58.860504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_clean_scrub_stamp":"2024-03-16T09:35:24.153508+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:43:06.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810201+0000","last_change":"2024-03-16T09:39:56.197109+0000","last_active":"2024-03-16T09:41:27.810201+0000","last_peered":"2024-03-16T09:41:27.810201+0000","last_clean":"2024-03-16T09:41:27.810201+0000","last_became_active":"2024-03-16T09:39:56.196775+0000","last_became_peered":"2024-03-16T09:39:56.196775+0000","last_unstale":"2024-03-16T09:41:27.810201+0000","last_undegraded":"2024-03-16T09:41:27.810201+0000","last_fullsized":"2024-03-16T09:41:27.810201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:29:10.401423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860552+0000","last_change":"2024-03-16T09:39:55.175744+0000","last_active":"2024-03-16T09:40:58.860552+0000","last_peered":"2024-03-16T09:40:58.860552+0000","last_clean":"2024-03-16T09:40:58.860552+0000","last_became_active":"2024-03-16T09:39:55.175445+0000","last_became_peered":"2024-03-16T09:39:55.175445+0000","last_unstale":"2024-03-16T09:40:58.860552+0000","last_undegraded":"2024-03-16T09:40:58.860552+0000","last_fullsized":"2024-03-16T09:40:58.860552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:39.494340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864068+0000","last_change":"2024-03-16T09:39:55.183441+0000","last_active":"2024-03-16T09:40:58.864068+0000","last_peered":"2024-03-16T09:40:58.864068+0000","last_clean":"2024-03-16T09:40:58.864068+0000","last_became_active":"2024-03-16T09:39:55.183115+0000","last_became_peered":"2024-03-16T09:39:55.183115+0000","last_unstale":"2024-03-16T09:40:58.864068+0000","last_undegraded":"2024-03-16T09:40:58.864068+0000","last_fullsized":"2024-03-16T09:40:58.864068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:55:31.525201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804957+0000","last_change":"2024-03-16T09:39:56.179205+0000","last_active":"2024-03-16T09:41:27.804957+0000","last_peered":"2024-03-16T09:41:27.804957+0000","last_clean":"2024-03-16T09:41:27.804957+0000","last_became_active":"2024-03-16T09:39:56.178849+0000","last_became_peered":"2024-03-16T09:39:56.178849+0000","last_unstale":"2024-03-16T09:41:27.804957+0000","last_undegraded":"2024-03-16T09:41:27.804957+0000","last_fullsized":"2024-03-16T09:41:27.804957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:10.016099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:28.195243+0000","last_change":"2024-03-16T09:39:55.180859+0000","last_active":"2024-03-16T09:41:28.195243+0000","last_peered":"2024-03-16T09:41:28.195243+0000","last_clean":"2024-03-16T09:41:28.195243+0000","last_became_active":"2024-03-16T09:39:55.180514+0000","last_became_peered":"2024-03-16T09:39:55.180514+0000","last_unstale":"2024-03-16T09:41:28.195243+0000","last_undegraded":"2024-03-16T09:41:28.195243+0000","last_fullsized":"2024-03-16T09:41:28.195243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:46.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806604+0000","last_change":"2024-03-16T09:39:56.187626+0000","last_active":"2024-03-16T09:41:27.806604+0000","last_peered":"2024-03-16T09:41:27.806604+0000","last_clean":"2024-03-16T09:41:27.806604+0000","last_became_active":"2024-03-16T09:39:56.187284+0000","last_became_peered":"2024-03-16T09:39:56.187284+0000","last_unstale":"2024-03-16T09:41:27.806604+0000","last_undegraded":"2024-03-16T09:41:27.806604+0000","last_fullsized":"2024-03-16T09:41:27.806604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:23.511603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864099+0000","last_change":"2024-03-16T09:39:55.172912+0000","last_active":"2024-03-16T09:40:58.864099+0000","last_peered":"2024-03-16T09:40:58.864099+0000","last_clean":"2024-03-16T09:40:58.864099+0000","last_became_active":"2024-03-16T09:39:55.172592+0000","last_became_peered":"2024-03-16T09:39:55.172592+0000","last_unstale":"2024-03-16T09:40:58.864099+0000","last_undegraded":"2024-03-16T09:40:58.864099+0000","last_fullsized":"2024-03-16T09:40:58.864099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:24.344588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811086+0000","last_change":"2024-03-16T09:39:56.197652+0000","last_active":"2024-03-16T09:41:27.811086+0000","last_peered":"2024-03-16T09:41:27.811086+0000","last_clean":"2024-03-16T09:41:27.811086+0000","last_became_active":"2024-03-16T09:39:56.197230+0000","last_became_peered":"2024-03-16T09:39:56.197230+0000","last_unstale":"2024-03-16T09:41:27.811086+0000","last_undegraded":"2024-03-16T09:41:27.811086+0000","last_fullsized":"2024-03-16T09:41:27.811086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:45:08.661372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.869780+0000","last_change":"2024-03-16T09:40:58.869780+0000","last_active":"2024-03-16T09:40:58.869780+0000","last_peered":"2024-03-16T09:40:58.869780+0000","last_clean":"2024-03-16T09:40:58.869780+0000","last_became_active":"2024-03-16T09:40:58.866452+0000","last_became_peered":"2024-03-16T09:40:58.866452+0000","last_unstale":"2024-03-16T09:40:58.869780+0000","last_undegraded":"2024-03-16T09:40:58.869780+0000","last_fullsized":"2024-03-16T09:40:58.869780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:07:21.751383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.771232+0000","last_change":"2024-03-16T09:39:56.179326+0000","last_active":"2024-03-16T09:41:27.771232+0000","last_peered":"2024-03-16T09:41:27.771232+0000","last_clean":"2024-03-16T09:41:27.771232+0000","last_became_active":"2024-03-16T09:39:56.178681+0000","last_became_peered":"2024-03-16T09:39:56.178681+0000","last_unstale":"2024-03-16T09:41:27.771232+0000","last_undegraded":"2024-03-16T09:41:27.771232+0000","last_fullsized":"2024-03-16T09:41:27.771232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:26.137807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871338+0000","last_change":"2024-03-16T09:39:55.173820+0000","last_active":"2024-03-16T09:40:58.871338+0000","last_peered":"2024-03-16T09:40:58.871338+0000","last_clean":"2024-03-16T09:40:58.871338+0000","last_became_active":"2024-03-16T09:39:55.173472+0000","last_became_peered":"2024-03-16T09:39:55.173472+0000","last_unstale":"2024-03-16T09:40:58.871338+0000","last_undegraded":"2024-03-16T09:40:58.871338+0000","last_fullsized":"2024-03-16T09:40:58.871338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:14.082367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802385+0000","last_change":"2024-03-16T09:39:56.178535+0000","last_active":"2024-03-16T09:41:27.802385+0000","last_peered":"2024-03-16T09:41:27.802385+0000","last_clean":"2024-03-16T09:41:27.802385+0000","last_became_active":"2024-03-16T09:39:56.178210+0000","last_became_peered":"2024-03-16T09:39:56.178210+0000","last_unstale":"2024-03-16T09:41:27.802385+0000","last_undegraded":"2024-03-16T09:41:27.802385+0000","last_fullsized":"2024-03-16T09:41:27.802385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:18.680469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864978+0000","last_change":"2024-03-16T09:39:55.183074+0000","last_active":"2024-03-16T09:40:58.864978+0000","last_peered":"2024-03-16T09:40:58.864978+0000","last_clean":"2024-03-16T09:40:58.864978+0000","last_became_active":"2024-03-16T09:39:55.182695+0000","last_became_peered":"2024-03-16T09:39:55.182695+0000","last_unstale":"2024-03-16T09:40:58.864978+0000","last_undegraded":"2024-03-16T09:40:58.864978+0000","last_fullsized":"2024-03-16T09:40:58.864978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:51.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804379+0000","last_change":"2024-03-16T09:39:56.193434+0000","last_active":"2024-03-16T09:41:27.804379+0000","last_peered":"2024-03-16T09:41:27.804379+0000","last_clean":"2024-03-16T09:41:27.804379+0000","last_became_active":"2024-03-16T09:39:56.193051+0000","last_became_peered":"2024-03-16T09:39:56.193051+0000","last_unstale":"2024-03-16T09:41:27.804379+0000","last_undegraded":"2024-03-16T09:41:27.804379+0000","last_fullsized":"2024-03-16T09:41:27.804379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:12:26.767117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863710+0000","last_change":"2024-03-16T09:39:55.179916+0000","last_active":"2024-03-16T09:40:58.863710+0000","last_peered":"2024-03-16T09:40:58.863710+0000","last_clean":"2024-03-16T09:40:58.863710+0000","last_became_active":"2024-03-16T09:39:55.179635+0000","last_became_peered":"2024-03-16T09:39:55.179635+0000","last_unstale":"2024-03-16T09:40:58.863710+0000","last_undegraded":"2024-03-16T09:40:58.863710+0000","last_fullsized":"2024-03-16T09:40:58.863710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:08.955039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814067+0000","last_change":"2024-03-16T09:39:56.194044+0000","last_active":"2024-03-16T09:41:27.814067+0000","last_peered":"2024-03-16T09:41:27.814067+0000","last_clean":"2024-03-16T09:41:27.814067+0000","last_became_active":"2024-03-16T09:39:56.193490+0000","last_became_peered":"2024-03-16T09:39:56.193490+0000","last_unstale":"2024-03-16T09:41:27.814067+0000","last_undegraded":"2024-03-16T09:41:27.814067+0000","last_fullsized":"2024-03-16T09:41:27.814067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:19:59.916892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865495+0000","last_change":"2024-03-16T09:39:55.182652+0000","last_active":"2024-03-16T09:40:58.865495+0000","last_peered":"2024-03-16T09:40:58.865495+0000","last_clean":"2024-03-16T09:40:58.865495+0000","last_became_active":"2024-03-16T09:39:55.181432+0000","last_became_peered":"2024-03-16T09:39:55.181432+0000","last_unstale":"2024-03-16T09:40:58.865495+0000","last_undegraded":"2024-03-16T09:40:58.865495+0000","last_fullsized":"2024-03-16T09:40:58.865495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:34.959845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817361+0000","last_change":"2024-03-16T09:39:56.188970+0000","last_active":"2024-03-16T09:41:27.817361+0000","last_peered":"2024-03-16T09:41:27.817361+0000","last_clean":"2024-03-16T09:41:27.817361+0000","last_became_active":"2024-03-16T09:39:56.188607+0000","last_became_peered":"2024-03-16T09:39:56.188607+0000","last_unstale":"2024-03-16T09:41:27.817361+0000","last_undegraded":"2024-03-16T09:41:27.817361+0000","last_fullsized":"2024-03-16T09:41:27.817361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:39:24.532114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863497+0000","last_change":"2024-03-16T09:39:55.181908+0000","last_active":"2024-03-16T09:40:58.863497+0000","last_peered":"2024-03-16T09:40:58.863497+0000","last_clean":"2024-03-16T09:40:58.863497+0000","last_became_active":"2024-03-16T09:39:55.181055+0000","last_became_peered":"2024-03-16T09:39:55.181055+0000","last_unstale":"2024-03-16T09:40:58.863497+0000","last_undegraded":"2024-03-16T09:40:58.863497+0000","last_fullsized":"2024-03-16T09:40:58.863497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:38.256664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803448+0000","last_change":"2024-03-16T09:39:56.189743+0000","last_active":"2024-03-16T09:41:27.803448+0000","last_peered":"2024-03-16T09:41:27.803448+0000","last_clean":"2024-03-16T09:41:27.803448+0000","last_became_active":"2024-03-16T09:39:56.189339+0000","last_became_peered":"2024-03-16T09:39:56.189339+0000","last_unstale":"2024-03-16T09:41:27.803448+0000","last_undegraded":"2024-03-16T09:41:27.803448+0000","last_fullsized":"2024-03-16T09:41:27.803448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:54.694191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.874651+0000","last_change":"2024-03-16T09:40:58.874651+0000","last_active":"2024-03-16T09:40:58.874651+0000","last_peered":"2024-03-16T09:40:58.874651+0000","last_clean":"2024-03-16T09:40:58.874651+0000","last_became_active":"2024-03-16T09:40:58.872329+0000","last_became_peered":"2024-03-16T09:40:58.872329+0000","last_unstale":"2024-03-16T09:40:58.874651+0000","last_undegraded":"2024-03-16T09:40:58.874651+0000","last_fullsized":"2024-03-16T09:40:58.874651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:05.424197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808467+0000","last_change":"2024-03-16T09:39:56.181636+0000","last_active":"2024-03-16T09:41:27.808467+0000","last_peered":"2024-03-16T09:41:27.808467+0000","last_clean":"2024-03-16T09:41:27.808467+0000","last_became_active":"2024-03-16T09:39:56.181309+0000","last_became_peered":"2024-03-16T09:39:56.181309+0000","last_unstale":"2024-03-16T09:41:27.808467+0000","last_undegraded":"2024-03-16T09:41:27.808467+0000","last_fullsized":"2024-03-16T09:41:27.808467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:36.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802234+0000","last_change":"2024-03-16T09:39:56.186640+0000","last_active":"2024-03-16T09:41:27.802234+0000","last_peered":"2024-03-16T09:41:27.802234+0000","last_clean":"2024-03-16T09:41:27.802234+0000","last_became_active":"2024-03-16T09:39:56.186311+0000","last_became_peered":"2024-03-16T09:39:56.186311+0000","last_unstale":"2024-03-16T09:41:27.802234+0000","last_undegraded":"2024-03-16T09:41:27.802234+0000","last_fullsized":"2024-03-16T09:41:27.802234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:58:44.451068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803937+0000","last_change":"2024-03-16T09:39:56.196887+0000","last_active":"2024-03-16T09:41:27.803937+0000","last_peered":"2024-03-16T09:41:27.803937+0000","last_clean":"2024-03-16T09:41:27.803937+0000","last_became_active":"2024-03-16T09:39:56.196568+0000","last_became_peered":"2024-03-16T09:39:56.196568+0000","last_unstale":"2024-03-16T09:41:27.803937+0000","last_undegraded":"2024-03-16T09:41:27.803937+0000","last_fullsized":"2024-03-16T09:41:27.803937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:08:35.577630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804119+0000","last_change":"2024-03-16T09:39:56.186394+0000","last_active":"2024-03-16T09:41:27.804119+0000","last_peered":"2024-03-16T09:41:27.804119+0000","last_clean":"2024-03-16T09:41:27.804119+0000","last_became_active":"2024-03-16T09:39:56.186053+0000","last_became_peered":"2024-03-16T09:39:56.186053+0000","last_unstale":"2024-03-16T09:41:27.804119+0000","last_undegraded":"2024-03-16T09:41:27.804119+0000","last_fullsized":"2024-03-16T09:41:27.804119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:41.529601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811065+0000","last_change":"2024-03-16T09:39:56.197622+0000","last_active":"2024-03-16T09:41:27.811065+0000","last_peered":"2024-03-16T09:41:27.811065+0000","last_clean":"2024-03-16T09:41:27.811065+0000","last_became_active":"2024-03-16T09:39:56.196885+0000","last_became_peered":"2024-03-16T09:39:56.196885+0000","last_unstale":"2024-03-16T09:41:27.811065+0000","last_undegraded":"2024-03-16T09:41:27.811065+0000","last_fullsized":"2024-03-16T09:41:27.811065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:08:50.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811555+0000","last_change":"2024-03-16T09:40:58.868417+0000","last_active":"2024-03-16T09:41:27.811555+0000","last_peered":"2024-03-16T09:41:27.811555+0000","last_clean":"2024-03-16T09:41:27.811555+0000","last_became_active":"2024-03-16T09:40:58.868100+0000","last_became_peered":"2024-03-16T09:40:58.868100+0000","last_unstale":"2024-03-16T09:41:27.811555+0000","last_undegraded":"2024-03-16T09:41:27.811555+0000","last_fullsized":"2024-03-16T09:41:27.811555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:51.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.799783+0000","last_change":"2024-03-16T09:39:56.181262+0000","last_active":"2024-03-16T09:41:27.799783+0000","last_peered":"2024-03-16T09:41:27.799783+0000","last_clean":"2024-03-16T09:41:27.799783+0000","last_became_active":"2024-03-16T09:39:56.180916+0000","last_became_peered":"2024-03-16T09:39:56.180916+0000","last_unstale":"2024-03-16T09:41:27.799783+0000","last_undegraded":"2024-03-16T09:41:27.799783+0000","last_fullsized":"2024-03-16T09:41:27.799783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:34:45.056025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818672+0000","last_change":"2024-03-16T09:39:56.176621+0000","last_active":"2024-03-16T09:41:27.818672+0000","last_peered":"2024-03-16T09:41:27.818672+0000","last_clean":"2024-03-16T09:41:27.818672+0000","last_became_active":"2024-03-16T09:39:56.176296+0000","last_became_peered":"2024-03-16T09:39:56.176296+0000","last_unstale":"2024-03-16T09:41:27.818672+0000","last_undegraded":"2024-03-16T09:41:27.818672+0000","last_fullsized":"2024-03-16T09:41:27.818672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:54.673937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806225+0000","last_change":"2024-03-16T09:39:56.197689+0000","last_active":"2024-03-16T09:41:27.806225+0000","last_peered":"2024-03-16T09:41:27.806225+0000","last_clean":"2024-03-16T09:41:27.806225+0000","last_became_active":"2024-03-16T09:39:56.197343+0000","last_became_peered":"2024-03-16T09:39:56.197343+0000","last_unstale":"2024-03-16T09:41:27.806225+0000","last_undegraded":"2024-03-16T09:41:27.806225+0000","last_fullsized":"2024-03-16T09:41:27.806225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:28:10.218048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.799490+0000","last_change":"2024-03-16T09:39:56.193551+0000","last_active":"2024-03-16T09:41:27.799490+0000","last_peered":"2024-03-16T09:41:27.799490+0000","last_clean":"2024-03-16T09:41:27.799490+0000","last_became_active":"2024-03-16T09:39:56.192901+0000","last_became_peered":"2024-03-16T09:39:56.192901+0000","last_unstale":"2024-03-16T09:41:27.799490+0000","last_undegraded":"2024-03-16T09:41:27.799490+0000","last_fullsized":"2024-03-16T09:41:27.799490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:50:15.528370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808698+0000","last_change":"2024-03-16T09:39:56.188484+0000","last_active":"2024-03-16T09:41:27.808698+0000","last_peered":"2024-03-16T09:41:27.808698+0000","last_clean":"2024-03-16T09:41:27.808698+0000","last_became_active":"2024-03-16T09:39:56.188144+0000","last_became_peered":"2024-03-16T09:39:56.188144+0000","last_unstale":"2024-03-16T09:41:27.808698+0000","last_undegraded":"2024-03-16T09:41:27.808698+0000","last_fullsized":"2024-03-16T09:41:27.808698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:01:10.042765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818669+0000","last_change":"2024-03-16T09:39:56.187448+0000","last_active":"2024-03-16T09:41:27.818669+0000","last_peered":"2024-03-16T09:41:27.818669+0000","last_clean":"2024-03-16T09:41:27.818669+0000","last_became_active":"2024-03-16T09:39:56.186772+0000","last_became_peered":"2024-03-16T09:39:56.186772+0000","last_unstale":"2024-03-16T09:41:27.818669+0000","last_undegraded":"2024-03-16T09:41:27.818669+0000","last_fullsized":"2024-03-16T09:41:27.818669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:45:43.315011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814777+0000","last_change":"2024-03-16T09:39:56.191004+0000","last_active":"2024-03-16T09:41:27.814777+0000","last_peered":"2024-03-16T09:41:27.814777+0000","last_clean":"2024-03-16T09:41:27.814777+0000","last_became_active":"2024-03-16T09:39:56.190646+0000","last_became_peered":"2024-03-16T09:39:56.190646+0000","last_unstale":"2024-03-16T09:41:27.814777+0000","last_undegraded":"2024-03-16T09:41:27.814777+0000","last_fullsized":"2024-03-16T09:41:27.814777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:03.191129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803969+0000","last_change":"2024-03-16T09:39:56.197552+0000","last_active":"2024-03-16T09:41:27.803969+0000","last_peered":"2024-03-16T09:41:27.803969+0000","last_clean":"2024-03-16T09:41:27.803969+0000","last_became_active":"2024-03-16T09:39:56.197215+0000","last_became_peered":"2024-03-16T09:39:56.197215+0000","last_unstale":"2024-03-16T09:41:27.803969+0000","last_undegraded":"2024-03-16T09:41:27.803969+0000","last_fullsized":"2024-03-16T09:41:27.803969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:45.795328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64868,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":690,"num_read_kb":416,"num_write":169,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":460713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1113430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037812,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":460825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1053372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":460316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":520874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":461587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":522717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":461246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36876,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708276,"statfs":{"total":95995035648,"available":95957274624,"internally_reserved":0,"allocated":1576960,"data_stored":1052070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T09:41:53.321 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-16T09:41:53.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T09:41:53.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 -- 172.21.15.77:0/3735486117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 msgr2=0x7fe1e8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 --2- 172.21.15.77:0/3735486117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e8108b40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009900 tx=0x7fe1d802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 -- 172.21.15.77:0/3735486117 shutdown_connections 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 --2- 172.21.15.77:0/3735486117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e8108b40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 --2- 172.21.15.77:0/3735486117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1e81018f0 0x7fe1e8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 --2- 172.21.15.77:0/3735486117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8100700 0x7fe1e8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 -- 172.21.15.77:0/3735486117 >> 172.21.15.77:0/3735486117 conn(0x7fe1e80fbc70 msgr2=0x7fe1e80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 -- 172.21.15.77:0/3735486117 shutdown_connections 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.455+0000 7fe1ed976640 1 -- 172.21.15.77:0/3735486117 wait complete. 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 Processor -- start 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 -- start start 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1e8100700 0x7fe1e8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e81018f0 0x7fe1e8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810c210 con 0x7fe1e81022b0 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810c090 con 0x7fe1e8100700 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1ed976640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810c390 con 0x7fe1e81018f0 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.456+0000 7fe1e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37064/0 (socket says 172.21.15.77:37064) 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 -- 172.21.15.77:0/2048898193 learned_addr learned my addr 172.21.15.77:0/2048898193 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e67fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e81018f0 0x7fe1e8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e6ffd640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1e8100700 0x7fe1e8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 -- 172.21.15.77:0/2048898193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e81018f0 msgr2=0x7fe1e8197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e81018f0 0x7fe1e8197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 -- 172.21.15.77:0/2048898193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1e8100700 msgr2=0x7fe1e8197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1e8100700 0x7fe1e8197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 -- 172.21.15.77:0/2048898193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e819f610 con 0x7fe1e81022b0 2024-03-16T09:41:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e67fc640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e81018f0 0x7fe1e8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1e77fe640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e819efa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009230 tx=0x7fe1d8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1ec974640 1 -- 172.21.15.77:0/2048898193 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8043020 con 0x7fe1e81022b0 2024-03-16T09:41:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e819f8a0 con 0x7fe1e81022b0 2024-03-16T09:41:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1ec974640 1 -- 172.21.15.77:0/2048898193 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1d80043d0 con 0x7fe1e81022b0 2024-03-16T09:41:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1ec974640 1 -- 172.21.15.77:0/2048898193 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8003b40 con 0x7fe1e81022b0 2024-03-16T09:41:53.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.457+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e819fe40 con 0x7fe1e81022b0 2024-03-16T09:41:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.458+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e81a0040 con 0x7fe1e81022b0 2024-03-16T09:41:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.459+0000 7fe1ec974640 1 -- 172.21.15.77:0/2048898193 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe1d8046020 con 0x7fe1e81022b0 2024-03-16T09:41:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.460+0000 7fe1ec974640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe1d00a2800 0x7fe1d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.460+0000 7fe1e6ffd640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe1d00a2800 0x7fe1d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.460+0000 7fe1ec974640 1 -- 172.21.15.77:0/2048898193 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe1d80f0bd0 con 0x7fe1e81022b0 2024-03-16T09:41:53.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.460+0000 7fe1e6ffd640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe1d00a2800 0x7fe1d00a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8101750 tx=0x7fe1e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:53.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.464+0000 7fe1ec974640 1 -- 172.21.15.77:0/2048898193 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1d804d020 con 0x7fe1e81022b0 2024-03-16T09:41:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.635+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 --> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1e80fde50 con 0x7fe1d00a2800 2024-03-16T09:41:53.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.642+0000 7fe1ec974640 1 -- 172.21.15.77:0/2048898193 <== mgr.14152 v2:172.21.15.77:6800/3114858521 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258804 (secure 0 0 0) 0x7fe1e80fde50 con 0x7fe1d00a2800 2024-03-16T09:41:53.648 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:53.650 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T09:41:53.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe1d00a2800 msgr2=0x7fe1d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe1d00a2800 0x7fe1d00a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8101750 tx=0x7fe1e0005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:53.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 msgr2=0x7fe1e819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e819efa0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009230 tx=0x7fe1d8005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:53.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 shutdown_connections 2024-03-16T09:41:53.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe1d00a2800 0x7fe1d00a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1e81022b0 0x7fe1e819efa0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e81018f0 0x7fe1e8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 --2- 172.21.15.77:0/2048898193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1e8100700 0x7fe1e8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 >> 172.21.15.77:0/2048898193 conn(0x7fe1e80fbc70 msgr2=0x7fe1e80fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:53.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 shutdown_connections 2024-03-16T09:41:53.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.645+0000 7fe1ed976640 1 -- 172.21.15.77:0/2048898193 wait complete. 2024-03-16T09:41:53.659 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-16T09:41:51.800371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655470,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":816,"num_read_kb":525,"num_write":352,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358300,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583524,"statfs":{"total":1151940427776,"available":1151573528576,"internally_reserved":0,"allocated":13627392,"data_stored":7488704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":986,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003299"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.812919+0000","last_change":"2024-03-16T09:39:56.184389+0000","last_active":"2024-03-16T09:41:27.812919+0000","last_peered":"2024-03-16T09:41:27.812919+0000","last_clean":"2024-03-16T09:41:27.812919+0000","last_became_active":"2024-03-16T09:39:56.184129+0000","last_became_peered":"2024-03-16T09:39:56.184129+0000","last_unstale":"2024-03-16T09:41:27.812919+0000","last_undegraded":"2024-03-16T09:41:27.812919+0000","last_fullsized":"2024-03-16T09:41:27.812919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:33.786788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.800995+0000","last_change":"2024-03-16T09:39:56.193451+0000","last_active":"2024-03-16T09:41:27.800995+0000","last_peered":"2024-03-16T09:41:27.800995+0000","last_clean":"2024-03-16T09:41:27.800995+0000","last_became_active":"2024-03-16T09:39:56.187381+0000","last_became_peered":"2024-03-16T09:39:56.187381+0000","last_unstale":"2024-03-16T09:41:27.800995+0000","last_undegraded":"2024-03-16T09:41:27.800995+0000","last_fullsized":"2024-03-16T09:41:27.800995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:51.915654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816523+0000","last_change":"2024-03-16T09:39:56.183785+0000","last_active":"2024-03-16T09:41:27.816523+0000","last_peered":"2024-03-16T09:41:27.816523+0000","last_clean":"2024-03-16T09:41:27.816523+0000","last_became_active":"2024-03-16T09:39:56.183421+0000","last_became_peered":"2024-03-16T09:39:56.183421+0000","last_unstale":"2024-03-16T09:41:27.816523+0000","last_undegraded":"2024-03-16T09:41:27.816523+0000","last_fullsized":"2024-03-16T09:41:27.816523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:47.259139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806416+0000","last_change":"2024-03-16T09:39:56.195743+0000","last_active":"2024-03-16T09:41:27.806416+0000","last_peered":"2024-03-16T09:41:27.806416+0000","last_clean":"2024-03-16T09:41:27.806416+0000","last_became_active":"2024-03-16T09:39:56.194818+0000","last_became_peered":"2024-03-16T09:39:56.194818+0000","last_unstale":"2024-03-16T09:41:27.806416+0000","last_undegraded":"2024-03-16T09:41:27.806416+0000","last_fullsized":"2024-03-16T09:41:27.806416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:39:43.123879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.809286+0000","last_change":"2024-03-16T09:39:56.184970+0000","last_active":"2024-03-16T09:41:27.809286+0000","last_peered":"2024-03-16T09:41:27.809286+0000","last_clean":"2024-03-16T09:41:27.809286+0000","last_became_active":"2024-03-16T09:39:56.184632+0000","last_became_peered":"2024-03-16T09:39:56.184632+0000","last_unstale":"2024-03-16T09:41:27.809286+0000","last_undegraded":"2024-03-16T09:41:27.809286+0000","last_fullsized":"2024-03-16T09:41:27.809286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:46:51.014402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804440+0000","last_change":"2024-03-16T09:39:56.199688+0000","last_active":"2024-03-16T09:41:27.804440+0000","last_peered":"2024-03-16T09:41:27.804440+0000","last_clean":"2024-03-16T09:41:27.804440+0000","last_became_active":"2024-03-16T09:39:56.199094+0000","last_became_peered":"2024-03-16T09:39:56.199094+0000","last_unstale":"2024-03-16T09:41:27.804440+0000","last_undegraded":"2024-03-16T09:41:27.804440+0000","last_fullsized":"2024-03-16T09:41:27.804440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:10.855504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.809387+0000","last_change":"2024-03-16T09:39:56.180816+0000","last_active":"2024-03-16T09:41:27.809387+0000","last_peered":"2024-03-16T09:41:27.809387+0000","last_clean":"2024-03-16T09:41:27.809387+0000","last_became_active":"2024-03-16T09:39:56.180499+0000","last_became_peered":"2024-03-16T09:39:56.180499+0000","last_unstale":"2024-03-16T09:41:27.809387+0000","last_undegraded":"2024-03-16T09:41:27.809387+0000","last_fullsized":"2024-03-16T09:41:27.809387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:32:09.444494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802950+0000","last_change":"2024-03-16T09:39:56.189960+0000","last_active":"2024-03-16T09:41:27.802950+0000","last_peered":"2024-03-16T09:41:27.802950+0000","last_clean":"2024-03-16T09:41:27.802950+0000","last_became_active":"2024-03-16T09:39:56.189609+0000","last_became_peered":"2024-03-16T09:39:56.189609+0000","last_unstale":"2024-03-16T09:41:27.802950+0000","last_undegraded":"2024-03-16T09:41:27.802950+0000","last_fullsized":"2024-03-16T09:41:27.802950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:59:50.219703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816982+0000","last_change":"2024-03-16T09:40:58.867996+0000","last_active":"2024-03-16T09:41:27.816982+0000","last_peered":"2024-03-16T09:41:27.816982+0000","last_clean":"2024-03-16T09:41:27.816982+0000","last_became_active":"2024-03-16T09:40:58.867619+0000","last_became_peered":"2024-03-16T09:40:58.867619+0000","last_unstale":"2024-03-16T09:41:27.816982+0000","last_undegraded":"2024-03-16T09:41:27.816982+0000","last_fullsized":"2024-03-16T09:41:27.816982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:36.972983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804258+0000","last_change":"2024-03-16T09:39:56.201584+0000","last_active":"2024-03-16T09:41:27.804258+0000","last_peered":"2024-03-16T09:41:27.804258+0000","last_clean":"2024-03-16T09:41:27.804258+0000","last_became_active":"2024-03-16T09:39:56.201320+0000","last_became_peered":"2024-03-16T09:39:56.201320+0000","last_unstale":"2024-03-16T09:41:27.804258+0000","last_undegraded":"2024-03-16T09:41:27.804258+0000","last_fullsized":"2024-03-16T09:41:27.804258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:18:00.564138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817302+0000","last_change":"2024-03-16T09:39:56.191125+0000","last_active":"2024-03-16T09:41:27.817302+0000","last_peered":"2024-03-16T09:41:27.817302+0000","last_clean":"2024-03-16T09:41:27.817302+0000","last_became_active":"2024-03-16T09:39:56.190795+0000","last_became_peered":"2024-03-16T09:39:56.190795+0000","last_unstale":"2024-03-16T09:41:27.817302+0000","last_undegraded":"2024-03-16T09:41:27.817302+0000","last_fullsized":"2024-03-16T09:41:27.817302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:44:10.894995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805793+0000","last_change":"2024-03-16T09:39:56.195652+0000","last_active":"2024-03-16T09:41:27.805793+0000","last_peered":"2024-03-16T09:41:27.805793+0000","last_clean":"2024-03-16T09:41:27.805793+0000","last_became_active":"2024-03-16T09:39:56.195270+0000","last_became_peered":"2024-03-16T09:39:56.195270+0000","last_unstale":"2024-03-16T09:41:27.805793+0000","last_undegraded":"2024-03-16T09:41:27.805793+0000","last_fullsized":"2024-03-16T09:41:27.805793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:03:41.546991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810598+0000","last_change":"2024-03-16T09:39:56.184622+0000","last_active":"2024-03-16T09:41:27.810598+0000","last_peered":"2024-03-16T09:41:27.810598+0000","last_clean":"2024-03-16T09:41:27.810598+0000","last_became_active":"2024-03-16T09:39:56.183532+0000","last_became_peered":"2024-03-16T09:39:56.183532+0000","last_unstale":"2024-03-16T09:41:27.810598+0000","last_undegraded":"2024-03-16T09:41:27.810598+0000","last_fullsized":"2024-03-16T09:41:27.810598+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:11:24.574088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808922+0000","last_change":"2024-03-16T09:39:56.184393+0000","last_active":"2024-03-16T09:41:27.808922+0000","last_peered":"2024-03-16T09:41:27.808922+0000","last_clean":"2024-03-16T09:41:27.808922+0000","last_became_active":"2024-03-16T09:39:56.183783+0000","last_became_peered":"2024-03-16T09:39:56.183783+0000","last_unstale":"2024-03-16T09:41:27.808922+0000","last_undegraded":"2024-03-16T09:41:27.808922+0000","last_fullsized":"2024-03-16T09:41:27.808922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:16:22.131116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811183+0000","last_change":"2024-03-16T09:39:56.197365+0000","last_active":"2024-03-16T09:41:27.811183+0000","last_peered":"2024-03-16T09:41:27.811183+0000","last_clean":"2024-03-16T09:41:27.811183+0000","last_became_active":"2024-03-16T09:39:56.197020+0000","last_became_peered":"2024-03-16T09:39:56.197020+0000","last_unstale":"2024-03-16T09:41:27.811183+0000","last_undegraded":"2024-03-16T09:41:27.811183+0000","last_fullsized":"2024-03-16T09:41:27.811183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:37:29.014062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817996+0000","last_change":"2024-03-16T09:39:56.185513+0000","last_active":"2024-03-16T09:41:27.817996+0000","last_peered":"2024-03-16T09:41:27.817996+0000","last_clean":"2024-03-16T09:41:27.817996+0000","last_became_active":"2024-03-16T09:39:56.184384+0000","last_became_peered":"2024-03-16T09:39:56.184384+0000","last_unstale":"2024-03-16T09:41:27.817996+0000","last_undegraded":"2024-03-16T09:41:27.817996+0000","last_fullsized":"2024-03-16T09:41:27.817996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:29.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.813059+0000","last_change":"2024-03-16T09:39:56.184536+0000","last_active":"2024-03-16T09:41:27.813059+0000","last_peered":"2024-03-16T09:41:27.813059+0000","last_clean":"2024-03-16T09:41:27.813059+0000","last_became_active":"2024-03-16T09:39:56.184208+0000","last_became_peered":"2024-03-16T09:39:56.184208+0000","last_unstale":"2024-03-16T09:41:27.813059+0000","last_undegraded":"2024-03-16T09:41:27.813059+0000","last_fullsized":"2024-03-16T09:41:27.813059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:27:14.025625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.796187+0000","last_change":"2024-03-16T09:39:56.184331+0000","last_active":"2024-03-16T09:41:27.796187+0000","last_peered":"2024-03-16T09:41:27.796187+0000","last_clean":"2024-03-16T09:41:27.796187+0000","last_became_active":"2024-03-16T09:39:56.183919+0000","last_became_peered":"2024-03-16T09:39:56.183919+0000","last_unstale":"2024-03-16T09:41:27.796187+0000","last_undegraded":"2024-03-16T09:41:27.796187+0000","last_fullsized":"2024-03-16T09:41:27.796187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:59.692854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802842+0000","last_change":"2024-03-16T09:39:56.185607+0000","last_active":"2024-03-16T09:41:27.802842+0000","last_peered":"2024-03-16T09:41:27.802842+0000","last_clean":"2024-03-16T09:41:27.802842+0000","last_became_active":"2024-03-16T09:39:56.185278+0000","last_became_peered":"2024-03-16T09:39:56.185278+0000","last_unstale":"2024-03-16T09:41:27.802842+0000","last_undegraded":"2024-03-16T09:41:27.802842+0000","last_fullsized":"2024-03-16T09:41:27.802842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:49:20.418334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860105+0000","last_change":"2024-03-16T09:39:55.180852+0000","last_active":"2024-03-16T09:40:58.860105+0000","last_peered":"2024-03-16T09:40:58.860105+0000","last_clean":"2024-03-16T09:40:58.860105+0000","last_became_active":"2024-03-16T09:39:55.180496+0000","last_became_peered":"2024-03-16T09:39:55.180496+0000","last_unstale":"2024-03-16T09:40:58.860105+0000","last_undegraded":"2024-03-16T09:40:58.860105+0000","last_fullsized":"2024-03-16T09:40:58.860105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:15:28.548458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805721+0000","last_change":"2024-03-16T09:39:56.190110+0000","last_active":"2024-03-16T09:41:27.805721+0000","last_peered":"2024-03-16T09:41:27.805721+0000","last_clean":"2024-03-16T09:41:27.805721+0000","last_became_active":"2024-03-16T09:39:56.189711+0000","last_became_peered":"2024-03-16T09:39:56.189711+0000","last_unstale":"2024-03-16T09:41:27.805721+0000","last_undegraded":"2024-03-16T09:41:27.805721+0000","last_fullsized":"2024-03-16T09:41:27.805721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:17.353457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871621+0000","last_change":"2024-03-16T09:39:55.171142+0000","last_active":"2024-03-16T09:40:58.871621+0000","last_peered":"2024-03-16T09:40:58.871621+0000","last_clean":"2024-03-16T09:40:58.871621+0000","last_became_active":"2024-03-16T09:39:55.170793+0000","last_became_peered":"2024-03-16T09:39:55.170793+0000","last_unstale":"2024-03-16T09:40:58.871621+0000","last_undegraded":"2024-03-16T09:40:58.871621+0000","last_fullsized":"2024-03-16T09:40:58.871621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:05:36.605553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816268+0000","last_change":"2024-03-16T09:39:56.186802+0000","last_active":"2024-03-16T09:41:27.816268+0000","last_peered":"2024-03-16T09:41:27.816268+0000","last_clean":"2024-03-16T09:41:27.816268+0000","last_became_active":"2024-03-16T09:39:56.186398+0000","last_became_peered":"2024-03-16T09:39:56.186398+0000","last_unstale":"2024-03-16T09:41:27.816268+0000","last_undegraded":"2024-03-16T09:41:27.816268+0000","last_fullsized":"2024-03-16T09:41:27.816268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:05:56.209864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864171+0000","last_change":"2024-03-16T09:39:55.181894+0000","last_active":"2024-03-16T09:40:58.864171+0000","last_peered":"2024-03-16T09:40:58.864171+0000","last_clean":"2024-03-16T09:40:58.864171+0000","last_became_active":"2024-03-16T09:39:55.179434+0000","last_became_peered":"2024-03-16T09:39:55.179434+0000","last_unstale":"2024-03-16T09:40:58.864171+0000","last_undegraded":"2024-03-16T09:40:58.864171+0000","last_fullsized":"2024-03-16T09:40:58.864171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:02:16.153928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818117+0000","last_change":"2024-03-16T09:39:56.188464+0000","last_active":"2024-03-16T09:41:27.818117+0000","last_peered":"2024-03-16T09:41:27.818117+0000","last_clean":"2024-03-16T09:41:27.818117+0000","last_became_active":"2024-03-16T09:39:56.188125+0000","last_became_peered":"2024-03-16T09:39:56.188125+0000","last_unstale":"2024-03-16T09:41:27.818117+0000","last_undegraded":"2024-03-16T09:41:27.818117+0000","last_fullsized":"2024-03-16T09:41:27.818117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:46:49.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863856+0000","last_change":"2024-03-16T09:39:55.179379+0000","last_active":"2024-03-16T09:40:58.863856+0000","last_peered":"2024-03-16T09:40:58.863856+0000","last_clean":"2024-03-16T09:40:58.863856+0000","last_became_active":"2024-03-16T09:39:55.178755+0000","last_became_peered":"2024-03-16T09:39:55.178755+0000","last_unstale":"2024-03-16T09:40:58.863856+0000","last_undegraded":"2024-03-16T09:40:58.863856+0000","last_fullsized":"2024-03-16T09:40:58.863856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:13:07.131417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.805630+0000","last_change":"2024-03-16T09:39:56.196655+0000","last_active":"2024-03-16T09:41:27.805630+0000","last_peered":"2024-03-16T09:41:27.805630+0000","last_clean":"2024-03-16T09:41:27.805630+0000","last_became_active":"2024-03-16T09:39:56.196344+0000","last_became_peered":"2024-03-16T09:39:56.196344+0000","last_unstale":"2024-03-16T09:41:27.805630+0000","last_undegraded":"2024-03-16T09:41:27.805630+0000","last_fullsized":"2024-03-16T09:41:27.805630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:12:03.920319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":134,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:45.842807+0000","last_change":"2024-03-16T09:39:55.182012+0000","last_active":"2024-03-16T09:41:45.842807+0000","last_peered":"2024-03-16T09:41:45.842807+0000","last_clean":"2024-03-16T09:41:45.842807+0000","last_became_active":"2024-03-16T09:39:55.178651+0000","last_became_peered":"2024-03-16T09:39:55.178651+0000","last_unstale":"2024-03-16T09:41:45.842807+0000","last_undegraded":"2024-03-16T09:41:45.842807+0000","last_fullsized":"2024-03-16T09:41:45.842807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:50:19.089698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.795850+0000","last_change":"2024-03-16T09:39:56.179645+0000","last_active":"2024-03-16T09:41:27.795850+0000","last_peered":"2024-03-16T09:41:27.795850+0000","last_clean":"2024-03-16T09:41:27.795850+0000","last_became_active":"2024-03-16T09:39:56.179257+0000","last_became_peered":"2024-03-16T09:39:56.179257+0000","last_unstale":"2024-03-16T09:41:27.795850+0000","last_undegraded":"2024-03-16T09:41:27.795850+0000","last_fullsized":"2024-03-16T09:41:27.795850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:30:09.104630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:59.218444+0000","last_change":"2024-03-16T09:39:55.178257+0000","last_active":"2024-03-16T09:40:59.218444+0000","last_peered":"2024-03-16T09:40:59.218444+0000","last_clean":"2024-03-16T09:40:59.218444+0000","last_became_active":"2024-03-16T09:39:55.177984+0000","last_became_peered":"2024-03-16T09:39:55.177984+0000","last_unstale":"2024-03-16T09:40:59.218444+0000","last_undegraded":"2024-03-16T09:40:59.218444+0000","last_fullsized":"2024-03-16T09:40:59.218444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.385624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804510+0000","last_change":"2024-03-16T09:39:56.196821+0000","last_active":"2024-03-16T09:41:27.804510+0000","last_peered":"2024-03-16T09:41:27.804510+0000","last_clean":"2024-03-16T09:41:27.804510+0000","last_became_active":"2024-03-16T09:39:56.196522+0000","last_became_peered":"2024-03-16T09:39:56.196522+0000","last_unstale":"2024-03-16T09:41:27.804510+0000","last_undegraded":"2024-03-16T09:41:27.804510+0000","last_fullsized":"2024-03-16T09:41:27.804510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:46:31.103530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865998+0000","last_change":"2024-03-16T09:39:55.170290+0000","last_active":"2024-03-16T09:40:58.865998+0000","last_peered":"2024-03-16T09:40:58.865998+0000","last_clean":"2024-03-16T09:40:58.865998+0000","last_became_active":"2024-03-16T09:39:55.169956+0000","last_became_peered":"2024-03-16T09:39:55.169956+0000","last_unstale":"2024-03-16T09:40:58.865998+0000","last_undegraded":"2024-03-16T09:40:58.865998+0000","last_fullsized":"2024-03-16T09:40:58.865998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:01:31.606917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.801687+0000","last_change":"2024-03-16T09:39:56.189155+0000","last_active":"2024-03-16T09:41:27.801687+0000","last_peered":"2024-03-16T09:41:27.801687+0000","last_clean":"2024-03-16T09:41:27.801687+0000","last_became_active":"2024-03-16T09:39:56.188827+0000","last_became_peered":"2024-03-16T09:39:56.188827+0000","last_unstale":"2024-03-16T09:41:27.801687+0000","last_undegraded":"2024-03-16T09:41:27.801687+0000","last_fullsized":"2024-03-16T09:41:27.801687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:03:54.370754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863093+0000","last_change":"2024-03-16T09:39:55.181886+0000","last_active":"2024-03-16T09:40:58.863093+0000","last_peered":"2024-03-16T09:40:58.863093+0000","last_clean":"2024-03-16T09:40:58.863093+0000","last_became_active":"2024-03-16T09:39:55.179836+0000","last_became_peered":"2024-03-16T09:39:55.179836+0000","last_unstale":"2024-03-16T09:40:58.863093+0000","last_undegraded":"2024-03-16T09:40:58.863093+0000","last_fullsized":"2024-03-16T09:40:58.863093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:52:38.772861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.795474+0000","last_change":"2024-03-16T09:39:56.185892+0000","last_active":"2024-03-16T09:41:27.795474+0000","last_peered":"2024-03-16T09:41:27.795474+0000","last_clean":"2024-03-16T09:41:27.795474+0000","last_became_active":"2024-03-16T09:39:56.185459+0000","last_became_peered":"2024-03-16T09:39:56.185459+0000","last_unstale":"2024-03-16T09:41:27.795474+0000","last_undegraded":"2024-03-16T09:41:27.795474+0000","last_fullsized":"2024-03-16T09:41:27.795474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:00:10.421054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":134,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:48.845916+0000","last_change":"2024-03-16T09:39:55.177607+0000","last_active":"2024-03-16T09:41:48.845916+0000","last_peered":"2024-03-16T09:41:48.845916+0000","last_clean":"2024-03-16T09:41:48.845916+0000","last_became_active":"2024-03-16T09:39:55.177265+0000","last_became_peered":"2024-03-16T09:39:55.177265+0000","last_unstale":"2024-03-16T09:41:48.845916+0000","last_undegraded":"2024-03-16T09:41:48.845916+0000","last_fullsized":"2024-03-16T09:41:48.845916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:39:47.849041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818050+0000","last_change":"2024-03-16T09:39:56.189172+0000","last_active":"2024-03-16T09:41:27.818050+0000","last_peered":"2024-03-16T09:41:27.818050+0000","last_clean":"2024-03-16T09:41:27.818050+0000","last_became_active":"2024-03-16T09:39:56.188798+0000","last_became_peered":"2024-03-16T09:39:56.188798+0000","last_unstale":"2024-03-16T09:41:27.818050+0000","last_undegraded":"2024-03-16T09:41:27.818050+0000","last_fullsized":"2024-03-16T09:41:27.818050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:02:11.119293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860780+0000","last_change":"2024-03-16T09:39:55.180369+0000","last_active":"2024-03-16T09:40:58.860780+0000","last_peered":"2024-03-16T09:40:58.860780+0000","last_clean":"2024-03-16T09:40:58.860780+0000","last_became_active":"2024-03-16T09:39:55.180053+0000","last_became_peered":"2024-03-16T09:39:55.180053+0000","last_unstale":"2024-03-16T09:40:58.860780+0000","last_undegraded":"2024-03-16T09:40:58.860780+0000","last_fullsized":"2024-03-16T09:40:58.860780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:53:44.990028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811041+0000","last_change":"2024-03-16T09:39:56.198454+0000","last_active":"2024-03-16T09:41:27.811041+0000","last_peered":"2024-03-16T09:41:27.811041+0000","last_clean":"2024-03-16T09:41:27.811041+0000","last_became_active":"2024-03-16T09:39:56.198097+0000","last_became_peered":"2024-03-16T09:39:56.198097+0000","last_unstale":"2024-03-16T09:41:27.811041+0000","last_undegraded":"2024-03-16T09:41:27.811041+0000","last_fullsized":"2024-03-16T09:41:27.811041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:12.494363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863700+0000","last_change":"2024-03-16T09:39:55.174221+0000","last_active":"2024-03-16T09:40:58.863700+0000","last_peered":"2024-03-16T09:40:58.863700+0000","last_clean":"2024-03-16T09:40:58.863700+0000","last_became_active":"2024-03-16T09:39:55.173920+0000","last_became_peered":"2024-03-16T09:39:55.173920+0000","last_unstale":"2024-03-16T09:40:58.863700+0000","last_undegraded":"2024-03-16T09:40:58.863700+0000","last_fullsized":"2024-03-16T09:40:58.863700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:36:25.108414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.816889+0000","last_change":"2024-03-16T09:39:56.196849+0000","last_active":"2024-03-16T09:41:27.816889+0000","last_peered":"2024-03-16T09:41:27.816889+0000","last_clean":"2024-03-16T09:41:27.816889+0000","last_became_active":"2024-03-16T09:39:56.196515+0000","last_became_peered":"2024-03-16T09:39:56.196515+0000","last_unstale":"2024-03-16T09:41:27.816889+0000","last_undegraded":"2024-03-16T09:41:27.816889+0000","last_fullsized":"2024-03-16T09:41:27.816889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:19:03.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":285,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:49.846725+0000","last_change":"2024-03-16T09:39:55.177788+0000","last_active":"2024-03-16T09:41:49.846725+0000","last_peered":"2024-03-16T09:41:49.846725+0000","last_clean":"2024-03-16T09:41:49.846725+0000","last_became_active":"2024-03-16T09:39:55.177443+0000","last_became_peered":"2024-03-16T09:39:55.177443+0000","last_unstale":"2024-03-16T09:41:49.846725+0000","last_undegraded":"2024-03-16T09:41:49.846725+0000","last_fullsized":"2024-03-16T09:41:49.846725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:44:39.221712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60765,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":232,"num_read_kb":175,"num_write":35,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.807296+0000","last_change":"2024-03-16T09:39:56.178976+0000","last_active":"2024-03-16T09:41:27.807296+0000","last_peered":"2024-03-16T09:41:27.807296+0000","last_clean":"2024-03-16T09:41:27.807296+0000","last_became_active":"2024-03-16T09:39:56.178455+0000","last_became_peered":"2024-03-16T09:39:56.178455+0000","last_unstale":"2024-03-16T09:41:27.807296+0000","last_undegraded":"2024-03-16T09:41:27.807296+0000","last_fullsized":"2024-03-16T09:41:27.807296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:52:08.064792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865191+0000","last_change":"2024-03-16T09:39:55.182579+0000","last_active":"2024-03-16T09:40:58.865191+0000","last_peered":"2024-03-16T09:40:58.865191+0000","last_clean":"2024-03-16T09:40:58.865191+0000","last_became_active":"2024-03-16T09:39:55.181298+0000","last_became_peered":"2024-03-16T09:39:55.181298+0000","last_unstale":"2024-03-16T09:40:58.865191+0000","last_undegraded":"2024-03-16T09:40:58.865191+0000","last_fullsized":"2024-03-16T09:40:58.865191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:35.768812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814563+0000","last_change":"2024-03-16T09:39:56.188519+0000","last_active":"2024-03-16T09:41:27.814563+0000","last_peered":"2024-03-16T09:41:27.814563+0000","last_clean":"2024-03-16T09:41:27.814563+0000","last_became_active":"2024-03-16T09:39:56.188031+0000","last_became_peered":"2024-03-16T09:39:56.188031+0000","last_unstale":"2024-03-16T09:41:27.814563+0000","last_undegraded":"2024-03-16T09:41:27.814563+0000","last_fullsized":"2024-03-16T09:41:27.814563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:20:31.765657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864304+0000","last_change":"2024-03-16T09:39:55.177523+0000","last_active":"2024-03-16T09:40:58.864304+0000","last_peered":"2024-03-16T09:40:58.864304+0000","last_clean":"2024-03-16T09:40:58.864304+0000","last_became_active":"2024-03-16T09:39:55.177143+0000","last_became_peered":"2024-03-16T09:39:55.177143+0000","last_unstale":"2024-03-16T09:40:58.864304+0000","last_undegraded":"2024-03-16T09:40:58.864304+0000","last_fullsized":"2024-03-16T09:40:58.864304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:05.357611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.813082+0000","last_change":"2024-03-16T09:40:58.867151+0000","last_active":"2024-03-16T09:41:27.813082+0000","last_peered":"2024-03-16T09:41:27.813082+0000","last_clean":"2024-03-16T09:41:27.813082+0000","last_became_active":"2024-03-16T09:40:58.866756+0000","last_became_peered":"2024-03-16T09:40:58.866756+0000","last_unstale":"2024-03-16T09:41:27.813082+0000","last_undegraded":"2024-03-16T09:41:27.813082+0000","last_fullsized":"2024-03-16T09:41:27.813082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:53.323964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:47.866162+0000","last_change":"2024-03-16T09:39:55.175419+0000","last_active":"2024-03-16T09:41:47.866162+0000","last_peered":"2024-03-16T09:41:47.866162+0000","last_clean":"2024-03-16T09:41:47.866162+0000","last_became_active":"2024-03-16T09:39:55.175126+0000","last_became_peered":"2024-03-16T09:39:55.175126+0000","last_unstale":"2024-03-16T09:41:47.866162+0000","last_undegraded":"2024-03-16T09:41:47.866162+0000","last_fullsized":"2024-03-16T09:41:47.866162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:11.696911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.812473+0000","last_change":"2024-03-16T09:39:56.176714+0000","last_active":"2024-03-16T09:41:27.812473+0000","last_peered":"2024-03-16T09:41:27.812473+0000","last_clean":"2024-03-16T09:41:27.812473+0000","last_became_active":"2024-03-16T09:39:56.176245+0000","last_became_peered":"2024-03-16T09:39:56.176245+0000","last_unstale":"2024-03-16T09:41:27.812473+0000","last_undegraded":"2024-03-16T09:41:27.812473+0000","last_fullsized":"2024-03-16T09:41:27.812473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:32.430207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863725+0000","last_change":"2024-03-16T09:39:55.181486+0000","last_active":"2024-03-16T09:40:58.863725+0000","last_peered":"2024-03-16T09:40:58.863725+0000","last_clean":"2024-03-16T09:40:58.863725+0000","last_became_active":"2024-03-16T09:39:55.180128+0000","last_became_peered":"2024-03-16T09:39:55.180128+0000","last_unstale":"2024-03-16T09:40:58.863725+0000","last_undegraded":"2024-03-16T09:40:58.863725+0000","last_fullsized":"2024-03-16T09:40:58.863725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:45.323870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803486+0000","last_change":"2024-03-16T09:39:56.196692+0000","last_active":"2024-03-16T09:41:27.803486+0000","last_peered":"2024-03-16T09:41:27.803486+0000","last_clean":"2024-03-16T09:41:27.803486+0000","last_became_active":"2024-03-16T09:39:56.196403+0000","last_became_peered":"2024-03-16T09:39:56.196403+0000","last_unstale":"2024-03-16T09:41:27.803486+0000","last_undegraded":"2024-03-16T09:41:27.803486+0000","last_fullsized":"2024-03-16T09:41:27.803486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:00:29.426476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871091+0000","last_change":"2024-03-16T09:40:58.870587+0000","last_active":"2024-03-16T09:40:58.871091+0000","last_peered":"2024-03-16T09:40:58.871091+0000","last_clean":"2024-03-16T09:40:58.871091+0000","last_became_active":"2024-03-16T09:40:58.864977+0000","last_became_peered":"2024-03-16T09:40:58.864977+0000","last_unstale":"2024-03-16T09:40:58.871091+0000","last_undegraded":"2024-03-16T09:40:58.871091+0000","last_fullsized":"2024-03-16T09:40:58.871091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:18.189307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.798760+0000","last_change":"2024-03-16T09:40:58.866074+0000","last_active":"2024-03-16T09:41:27.798760+0000","last_peered":"2024-03-16T09:41:27.798760+0000","last_clean":"2024-03-16T09:41:27.798760+0000","last_became_active":"2024-03-16T09:40:58.865711+0000","last_became_peered":"2024-03-16T09:40:58.865711+0000","last_unstale":"2024-03-16T09:41:27.798760+0000","last_undegraded":"2024-03-16T09:41:27.798760+0000","last_fullsized":"2024-03-16T09:41:27.798760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:59:57.137802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.859988+0000","last_change":"2024-03-16T09:39:55.175666+0000","last_active":"2024-03-16T09:40:58.859988+0000","last_peered":"2024-03-16T09:40:58.859988+0000","last_clean":"2024-03-16T09:40:58.859988+0000","last_became_active":"2024-03-16T09:39:55.175337+0000","last_became_peered":"2024-03-16T09:39:55.175337+0000","last_unstale":"2024-03-16T09:40:58.859988+0000","last_undegraded":"2024-03-16T09:40:58.859988+0000","last_fullsized":"2024-03-16T09:40:58.859988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:45:13.935669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810754+0000","last_change":"2024-03-16T09:39:56.184878+0000","last_active":"2024-03-16T09:41:27.810754+0000","last_peered":"2024-03-16T09:41:27.810754+0000","last_clean":"2024-03-16T09:41:27.810754+0000","last_became_active":"2024-03-16T09:39:56.184480+0000","last_became_peered":"2024-03-16T09:39:56.184480+0000","last_unstale":"2024-03-16T09:41:27.810754+0000","last_undegraded":"2024-03-16T09:41:27.810754+0000","last_fullsized":"2024-03-16T09:41:27.810754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:40:01.286993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860584+0000","last_change":"2024-03-16T09:39:55.179189+0000","last_active":"2024-03-16T09:40:58.860584+0000","last_peered":"2024-03-16T09:40:58.860584+0000","last_clean":"2024-03-16T09:40:58.860584+0000","last_became_active":"2024-03-16T09:39:55.178859+0000","last_became_peered":"2024-03-16T09:39:55.178859+0000","last_unstale":"2024-03-16T09:40:58.860584+0000","last_undegraded":"2024-03-16T09:40:58.860584+0000","last_fullsized":"2024-03-16T09:40:58.860584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:54:15.498386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.788321+0000","last_change":"2024-03-16T09:39:56.200665+0000","last_active":"2024-03-16T09:41:27.788321+0000","last_peered":"2024-03-16T09:41:27.788321+0000","last_clean":"2024-03-16T09:41:27.788321+0000","last_became_active":"2024-03-16T09:39:56.200326+0000","last_became_peered":"2024-03-16T09:39:56.200326+0000","last_unstale":"2024-03-16T09:41:27.788321+0000","last_undegraded":"2024-03-16T09:41:27.788321+0000","last_fullsized":"2024-03-16T09:41:27.788321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:30:34.778630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806467+0000","last_change":"2024-03-16T09:39:56.182670+0000","last_active":"2024-03-16T09:41:27.806467+0000","last_peered":"2024-03-16T09:41:27.806467+0000","last_clean":"2024-03-16T09:41:27.806467+0000","last_became_active":"2024-03-16T09:39:56.182009+0000","last_became_peered":"2024-03-16T09:39:56.182009+0000","last_unstale":"2024-03-16T09:41:27.806467+0000","last_undegraded":"2024-03-16T09:41:27.806467+0000","last_fullsized":"2024-03-16T09:41:27.806467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:28:03.106071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:49.869064+0000","last_change":"2024-03-16T09:39:55.171609+0000","last_active":"2024-03-16T09:41:49.869064+0000","last_peered":"2024-03-16T09:41:49.869064+0000","last_clean":"2024-03-16T09:41:49.869064+0000","last_became_active":"2024-03-16T09:39:55.171301+0000","last_became_peered":"2024-03-16T09:39:55.171301+0000","last_unstale":"2024-03-16T09:41:49.869064+0000","last_undegraded":"2024-03-16T09:41:49.869064+0000","last_fullsized":"2024-03-16T09:41:49.869064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:28.265342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.787638+0000","last_change":"2024-03-16T09:40:58.865311+0000","last_active":"2024-03-16T09:41:27.787638+0000","last_peered":"2024-03-16T09:41:27.787638+0000","last_clean":"2024-03-16T09:41:27.787638+0000","last_became_active":"2024-03-16T09:40:58.864918+0000","last_became_peered":"2024-03-16T09:40:58.864918+0000","last_unstale":"2024-03-16T09:41:27.787638+0000","last_undegraded":"2024-03-16T09:41:27.787638+0000","last_fullsized":"2024-03-16T09:41:27.787638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:55:45.162900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863236+0000","last_change":"2024-03-16T09:39:55.172824+0000","last_active":"2024-03-16T09:40:58.863236+0000","last_peered":"2024-03-16T09:40:58.863236+0000","last_clean":"2024-03-16T09:40:58.863236+0000","last_became_active":"2024-03-16T09:39:55.172493+0000","last_became_peered":"2024-03-16T09:39:55.172493+0000","last_unstale":"2024-03-16T09:40:58.863236+0000","last_undegraded":"2024-03-16T09:40:58.863236+0000","last_fullsized":"2024-03-16T09:40:58.863236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:39.188176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":346,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860504+0000","last_change":"2024-03-16T09:38:43.693644+0000","last_active":"2024-03-16T09:40:58.860504+0000","last_peered":"2024-03-16T09:40:58.860504+0000","last_clean":"2024-03-16T09:40:58.860504+0000","last_became_active":"2024-03-16T09:38:43.693202+0000","last_became_peered":"2024-03-16T09:38:43.693202+0000","last_unstale":"2024-03-16T09:40:58.860504+0000","last_undegraded":"2024-03-16T09:40:58.860504+0000","last_fullsized":"2024-03-16T09:40:58.860504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:35:24.153508+0000","last_clean_scrub_stamp":"2024-03-16T09:35:24.153508+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:43:06.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.810201+0000","last_change":"2024-03-16T09:39:56.197109+0000","last_active":"2024-03-16T09:41:27.810201+0000","last_peered":"2024-03-16T09:41:27.810201+0000","last_clean":"2024-03-16T09:41:27.810201+0000","last_became_active":"2024-03-16T09:39:56.196775+0000","last_became_peered":"2024-03-16T09:39:56.196775+0000","last_unstale":"2024-03-16T09:41:27.810201+0000","last_undegraded":"2024-03-16T09:41:27.810201+0000","last_fullsized":"2024-03-16T09:41:27.810201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:29:10.401423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.860552+0000","last_change":"2024-03-16T09:39:55.175744+0000","last_active":"2024-03-16T09:40:58.860552+0000","last_peered":"2024-03-16T09:40:58.860552+0000","last_clean":"2024-03-16T09:40:58.860552+0000","last_became_active":"2024-03-16T09:39:55.175445+0000","last_became_peered":"2024-03-16T09:39:55.175445+0000","last_unstale":"2024-03-16T09:40:58.860552+0000","last_undegraded":"2024-03-16T09:40:58.860552+0000","last_fullsized":"2024-03-16T09:40:58.860552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:39.494340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864068+0000","last_change":"2024-03-16T09:39:55.183441+0000","last_active":"2024-03-16T09:40:58.864068+0000","last_peered":"2024-03-16T09:40:58.864068+0000","last_clean":"2024-03-16T09:40:58.864068+0000","last_became_active":"2024-03-16T09:39:55.183115+0000","last_became_peered":"2024-03-16T09:39:55.183115+0000","last_unstale":"2024-03-16T09:40:58.864068+0000","last_undegraded":"2024-03-16T09:40:58.864068+0000","last_fullsized":"2024-03-16T09:40:58.864068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:55:31.525201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804957+0000","last_change":"2024-03-16T09:39:56.179205+0000","last_active":"2024-03-16T09:41:27.804957+0000","last_peered":"2024-03-16T09:41:27.804957+0000","last_clean":"2024-03-16T09:41:27.804957+0000","last_became_active":"2024-03-16T09:39:56.178849+0000","last_became_peered":"2024-03-16T09:39:56.178849+0000","last_unstale":"2024-03-16T09:41:27.804957+0000","last_undegraded":"2024-03-16T09:41:27.804957+0000","last_fullsized":"2024-03-16T09:41:27.804957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:10.016099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:28.195243+0000","last_change":"2024-03-16T09:39:55.180859+0000","last_active":"2024-03-16T09:41:28.195243+0000","last_peered":"2024-03-16T09:41:28.195243+0000","last_clean":"2024-03-16T09:41:28.195243+0000","last_became_active":"2024-03-16T09:39:55.180514+0000","last_became_peered":"2024-03-16T09:39:55.180514+0000","last_unstale":"2024-03-16T09:41:28.195243+0000","last_undegraded":"2024-03-16T09:41:28.195243+0000","last_fullsized":"2024-03-16T09:41:28.195243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:46.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806604+0000","last_change":"2024-03-16T09:39:56.187626+0000","last_active":"2024-03-16T09:41:27.806604+0000","last_peered":"2024-03-16T09:41:27.806604+0000","last_clean":"2024-03-16T09:41:27.806604+0000","last_became_active":"2024-03-16T09:39:56.187284+0000","last_became_peered":"2024-03-16T09:39:56.187284+0000","last_unstale":"2024-03-16T09:41:27.806604+0000","last_undegraded":"2024-03-16T09:41:27.806604+0000","last_fullsized":"2024-03-16T09:41:27.806604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:23.511603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864099+0000","last_change":"2024-03-16T09:39:55.172912+0000","last_active":"2024-03-16T09:40:58.864099+0000","last_peered":"2024-03-16T09:40:58.864099+0000","last_clean":"2024-03-16T09:40:58.864099+0000","last_became_active":"2024-03-16T09:39:55.172592+0000","last_became_peered":"2024-03-16T09:39:55.172592+0000","last_unstale":"2024-03-16T09:40:58.864099+0000","last_undegraded":"2024-03-16T09:40:58.864099+0000","last_fullsized":"2024-03-16T09:40:58.864099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:24.344588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811086+0000","last_change":"2024-03-16T09:39:56.197652+0000","last_active":"2024-03-16T09:41:27.811086+0000","last_peered":"2024-03-16T09:41:27.811086+0000","last_clean":"2024-03-16T09:41:27.811086+0000","last_became_active":"2024-03-16T09:39:56.197230+0000","last_became_peered":"2024-03-16T09:39:56.197230+0000","last_unstale":"2024-03-16T09:41:27.811086+0000","last_undegraded":"2024-03-16T09:41:27.811086+0000","last_fullsized":"2024-03-16T09:41:27.811086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:45:08.661372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.869780+0000","last_change":"2024-03-16T09:40:58.869780+0000","last_active":"2024-03-16T09:40:58.869780+0000","last_peered":"2024-03-16T09:40:58.869780+0000","last_clean":"2024-03-16T09:40:58.869780+0000","last_became_active":"2024-03-16T09:40:58.866452+0000","last_became_peered":"2024-03-16T09:40:58.866452+0000","last_unstale":"2024-03-16T09:40:58.869780+0000","last_undegraded":"2024-03-16T09:40:58.869780+0000","last_fullsized":"2024-03-16T09:40:58.869780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:07:21.751383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.771232+0000","last_change":"2024-03-16T09:39:56.179326+0000","last_active":"2024-03-16T09:41:27.771232+0000","last_peered":"2024-03-16T09:41:27.771232+0000","last_clean":"2024-03-16T09:41:27.771232+0000","last_became_active":"2024-03-16T09:39:56.178681+0000","last_became_peered":"2024-03-16T09:39:56.178681+0000","last_unstale":"2024-03-16T09:41:27.771232+0000","last_undegraded":"2024-03-16T09:41:27.771232+0000","last_fullsized":"2024-03-16T09:41:27.771232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:26.137807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.871338+0000","last_change":"2024-03-16T09:39:55.173820+0000","last_active":"2024-03-16T09:40:58.871338+0000","last_peered":"2024-03-16T09:40:58.871338+0000","last_clean":"2024-03-16T09:40:58.871338+0000","last_became_active":"2024-03-16T09:39:55.173472+0000","last_became_peered":"2024-03-16T09:39:55.173472+0000","last_unstale":"2024-03-16T09:40:58.871338+0000","last_undegraded":"2024-03-16T09:40:58.871338+0000","last_fullsized":"2024-03-16T09:40:58.871338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:13:14.082367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802385+0000","last_change":"2024-03-16T09:39:56.178535+0000","last_active":"2024-03-16T09:41:27.802385+0000","last_peered":"2024-03-16T09:41:27.802385+0000","last_clean":"2024-03-16T09:41:27.802385+0000","last_became_active":"2024-03-16T09:39:56.178210+0000","last_became_peered":"2024-03-16T09:39:56.178210+0000","last_unstale":"2024-03-16T09:41:27.802385+0000","last_undegraded":"2024-03-16T09:41:27.802385+0000","last_fullsized":"2024-03-16T09:41:27.802385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:08:18.680469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.864978+0000","last_change":"2024-03-16T09:39:55.183074+0000","last_active":"2024-03-16T09:40:58.864978+0000","last_peered":"2024-03-16T09:40:58.864978+0000","last_clean":"2024-03-16T09:40:58.864978+0000","last_became_active":"2024-03-16T09:39:55.182695+0000","last_became_peered":"2024-03-16T09:39:55.182695+0000","last_unstale":"2024-03-16T09:40:58.864978+0000","last_undegraded":"2024-03-16T09:40:58.864978+0000","last_fullsized":"2024-03-16T09:40:58.864978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:51.397190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804379+0000","last_change":"2024-03-16T09:39:56.193434+0000","last_active":"2024-03-16T09:41:27.804379+0000","last_peered":"2024-03-16T09:41:27.804379+0000","last_clean":"2024-03-16T09:41:27.804379+0000","last_became_active":"2024-03-16T09:39:56.193051+0000","last_became_peered":"2024-03-16T09:39:56.193051+0000","last_unstale":"2024-03-16T09:41:27.804379+0000","last_undegraded":"2024-03-16T09:41:27.804379+0000","last_fullsized":"2024-03-16T09:41:27.804379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:12:26.767117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863710+0000","last_change":"2024-03-16T09:39:55.179916+0000","last_active":"2024-03-16T09:40:58.863710+0000","last_peered":"2024-03-16T09:40:58.863710+0000","last_clean":"2024-03-16T09:40:58.863710+0000","last_became_active":"2024-03-16T09:39:55.179635+0000","last_became_peered":"2024-03-16T09:39:55.179635+0000","last_unstale":"2024-03-16T09:40:58.863710+0000","last_undegraded":"2024-03-16T09:40:58.863710+0000","last_fullsized":"2024-03-16T09:40:58.863710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:08.955039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814067+0000","last_change":"2024-03-16T09:39:56.194044+0000","last_active":"2024-03-16T09:41:27.814067+0000","last_peered":"2024-03-16T09:41:27.814067+0000","last_clean":"2024-03-16T09:41:27.814067+0000","last_became_active":"2024-03-16T09:39:56.193490+0000","last_became_peered":"2024-03-16T09:39:56.193490+0000","last_unstale":"2024-03-16T09:41:27.814067+0000","last_undegraded":"2024-03-16T09:41:27.814067+0000","last_fullsized":"2024-03-16T09:41:27.814067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:19:59.916892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.865495+0000","last_change":"2024-03-16T09:39:55.182652+0000","last_active":"2024-03-16T09:40:58.865495+0000","last_peered":"2024-03-16T09:40:58.865495+0000","last_clean":"2024-03-16T09:40:58.865495+0000","last_became_active":"2024-03-16T09:39:55.181432+0000","last_became_peered":"2024-03-16T09:39:55.181432+0000","last_unstale":"2024-03-16T09:40:58.865495+0000","last_undegraded":"2024-03-16T09:40:58.865495+0000","last_fullsized":"2024-03-16T09:40:58.865495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:34.959845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.817361+0000","last_change":"2024-03-16T09:39:56.188970+0000","last_active":"2024-03-16T09:41:27.817361+0000","last_peered":"2024-03-16T09:41:27.817361+0000","last_clean":"2024-03-16T09:41:27.817361+0000","last_became_active":"2024-03-16T09:39:56.188607+0000","last_became_peered":"2024-03-16T09:39:56.188607+0000","last_unstale":"2024-03-16T09:41:27.817361+0000","last_undegraded":"2024-03-16T09:41:27.817361+0000","last_fullsized":"2024-03-16T09:41:27.817361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:39:24.532114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.863497+0000","last_change":"2024-03-16T09:39:55.181908+0000","last_active":"2024-03-16T09:40:58.863497+0000","last_peered":"2024-03-16T09:40:58.863497+0000","last_clean":"2024-03-16T09:40:58.863497+0000","last_became_active":"2024-03-16T09:39:55.181055+0000","last_became_peered":"2024-03-16T09:39:55.181055+0000","last_unstale":"2024-03-16T09:40:58.863497+0000","last_undegraded":"2024-03-16T09:40:58.863497+0000","last_fullsized":"2024-03-16T09:40:58.863497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:28:38.256664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803448+0000","last_change":"2024-03-16T09:39:56.189743+0000","last_active":"2024-03-16T09:41:27.803448+0000","last_peered":"2024-03-16T09:41:27.803448+0000","last_clean":"2024-03-16T09:41:27.803448+0000","last_became_active":"2024-03-16T09:39:56.189339+0000","last_became_peered":"2024-03-16T09:39:56.189339+0000","last_unstale":"2024-03-16T09:41:27.803448+0000","last_undegraded":"2024-03-16T09:41:27.803448+0000","last_fullsized":"2024-03-16T09:41:27.803448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:54.694191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:40:58.874651+0000","last_change":"2024-03-16T09:40:58.874651+0000","last_active":"2024-03-16T09:40:58.874651+0000","last_peered":"2024-03-16T09:40:58.874651+0000","last_clean":"2024-03-16T09:40:58.874651+0000","last_became_active":"2024-03-16T09:40:58.872329+0000","last_became_peered":"2024-03-16T09:40:58.872329+0000","last_unstale":"2024-03-16T09:40:58.874651+0000","last_undegraded":"2024-03-16T09:40:58.874651+0000","last_fullsized":"2024-03-16T09:40:58.874651+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:54.151981+0000","last_clean_scrub_stamp":"2024-03-16T09:39:54.151981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:05.424197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808467+0000","last_change":"2024-03-16T09:39:56.181636+0000","last_active":"2024-03-16T09:41:27.808467+0000","last_peered":"2024-03-16T09:41:27.808467+0000","last_clean":"2024-03-16T09:41:27.808467+0000","last_became_active":"2024-03-16T09:39:56.181309+0000","last_became_peered":"2024-03-16T09:39:56.181309+0000","last_unstale":"2024-03-16T09:41:27.808467+0000","last_undegraded":"2024-03-16T09:41:27.808467+0000","last_fullsized":"2024-03-16T09:41:27.808467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:36.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.802234+0000","last_change":"2024-03-16T09:39:56.186640+0000","last_active":"2024-03-16T09:41:27.802234+0000","last_peered":"2024-03-16T09:41:27.802234+0000","last_clean":"2024-03-16T09:41:27.802234+0000","last_became_active":"2024-03-16T09:39:56.186311+0000","last_became_peered":"2024-03-16T09:39:56.186311+0000","last_unstale":"2024-03-16T09:41:27.802234+0000","last_undegraded":"2024-03-16T09:41:27.802234+0000","last_fullsized":"2024-03-16T09:41:27.802234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:58:44.451068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803937+0000","last_change":"2024-03-16T09:39:56.196887+0000","last_active":"2024-03-16T09:41:27.803937+0000","last_peered":"2024-03-16T09:41:27.803937+0000","last_clean":"2024-03-16T09:41:27.803937+0000","last_became_active":"2024-03-16T09:39:56.196568+0000","last_became_peered":"2024-03-16T09:39:56.196568+0000","last_unstale":"2024-03-16T09:41:27.803937+0000","last_undegraded":"2024-03-16T09:41:27.803937+0000","last_fullsized":"2024-03-16T09:41:27.803937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:08:35.577630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.804119+0000","last_change":"2024-03-16T09:39:56.186394+0000","last_active":"2024-03-16T09:41:27.804119+0000","last_peered":"2024-03-16T09:41:27.804119+0000","last_clean":"2024-03-16T09:41:27.804119+0000","last_became_active":"2024-03-16T09:39:56.186053+0000","last_became_peered":"2024-03-16T09:39:56.186053+0000","last_unstale":"2024-03-16T09:41:27.804119+0000","last_undegraded":"2024-03-16T09:41:27.804119+0000","last_fullsized":"2024-03-16T09:41:27.804119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:32:41.529601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811065+0000","last_change":"2024-03-16T09:39:56.197622+0000","last_active":"2024-03-16T09:41:27.811065+0000","last_peered":"2024-03-16T09:41:27.811065+0000","last_clean":"2024-03-16T09:41:27.811065+0000","last_became_active":"2024-03-16T09:39:56.196885+0000","last_became_peered":"2024-03-16T09:39:56.196885+0000","last_unstale":"2024-03-16T09:41:27.811065+0000","last_undegraded":"2024-03-16T09:41:27.811065+0000","last_fullsized":"2024-03-16T09:41:27.811065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:08:50.554049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.811555+0000","last_change":"2024-03-16T09:40:58.868417+0000","last_active":"2024-03-16T09:41:27.811555+0000","last_peered":"2024-03-16T09:41:27.811555+0000","last_clean":"2024-03-16T09:41:27.811555+0000","last_became_active":"2024-03-16T09:40:58.868100+0000","last_became_peered":"2024-03-16T09:40:58.868100+0000","last_unstale":"2024-03-16T09:41:27.811555+0000","last_undegraded":"2024-03-16T09:41:27.811555+0000","last_fullsized":"2024-03-16T09:41:27.811555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:51.545079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.799783+0000","last_change":"2024-03-16T09:39:56.181262+0000","last_active":"2024-03-16T09:41:27.799783+0000","last_peered":"2024-03-16T09:41:27.799783+0000","last_clean":"2024-03-16T09:41:27.799783+0000","last_became_active":"2024-03-16T09:39:56.180916+0000","last_became_peered":"2024-03-16T09:39:56.180916+0000","last_unstale":"2024-03-16T09:41:27.799783+0000","last_undegraded":"2024-03-16T09:41:27.799783+0000","last_fullsized":"2024-03-16T09:41:27.799783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:34:45.056025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818672+0000","last_change":"2024-03-16T09:39:56.176621+0000","last_active":"2024-03-16T09:41:27.818672+0000","last_peered":"2024-03-16T09:41:27.818672+0000","last_clean":"2024-03-16T09:41:27.818672+0000","last_became_active":"2024-03-16T09:39:56.176296+0000","last_became_peered":"2024-03-16T09:39:56.176296+0000","last_unstale":"2024-03-16T09:41:27.818672+0000","last_undegraded":"2024-03-16T09:41:27.818672+0000","last_fullsized":"2024-03-16T09:41:27.818672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:54.673937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.806225+0000","last_change":"2024-03-16T09:39:56.197689+0000","last_active":"2024-03-16T09:41:27.806225+0000","last_peered":"2024-03-16T09:41:27.806225+0000","last_clean":"2024-03-16T09:41:27.806225+0000","last_became_active":"2024-03-16T09:39:56.197343+0000","last_became_peered":"2024-03-16T09:39:56.197343+0000","last_unstale":"2024-03-16T09:41:27.806225+0000","last_undegraded":"2024-03-16T09:41:27.806225+0000","last_fullsized":"2024-03-16T09:41:27.806225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:28:10.218048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.799490+0000","last_change":"2024-03-16T09:39:56.193551+0000","last_active":"2024-03-16T09:41:27.799490+0000","last_peered":"2024-03-16T09:41:27.799490+0000","last_clean":"2024-03-16T09:41:27.799490+0000","last_became_active":"2024-03-16T09:39:56.192901+0000","last_became_peered":"2024-03-16T09:39:56.192901+0000","last_unstale":"2024-03-16T09:41:27.799490+0000","last_undegraded":"2024-03-16T09:41:27.799490+0000","last_fullsized":"2024-03-16T09:41:27.799490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:50:15.528370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.808698+0000","last_change":"2024-03-16T09:39:56.188484+0000","last_active":"2024-03-16T09:41:27.808698+0000","last_peered":"2024-03-16T09:41:27.808698+0000","last_clean":"2024-03-16T09:41:27.808698+0000","last_became_active":"2024-03-16T09:39:56.188144+0000","last_became_peered":"2024-03-16T09:39:56.188144+0000","last_unstale":"2024-03-16T09:41:27.808698+0000","last_undegraded":"2024-03-16T09:41:27.808698+0000","last_fullsized":"2024-03-16T09:41:27.808698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:01:10.042765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.818669+0000","last_change":"2024-03-16T09:39:56.187448+0000","last_active":"2024-03-16T09:41:27.818669+0000","last_peered":"2024-03-16T09:41:27.818669+0000","last_clean":"2024-03-16T09:41:27.818669+0000","last_became_active":"2024-03-16T09:39:56.186772+0000","last_became_peered":"2024-03-16T09:39:56.186772+0000","last_unstale":"2024-03-16T09:41:27.818669+0000","last_undegraded":"2024-03-16T09:41:27.818669+0000","last_fullsized":"2024-03-16T09:41:27.818669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T09:45:43.315011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.814777+0000","last_change":"2024-03-16T09:39:56.191004+0000","last_active":"2024-03-16T09:41:27.814777+0000","last_peered":"2024-03-16T09:41:27.814777+0000","last_clean":"2024-03-16T09:41:27.814777+0000","last_became_active":"2024-03-16T09:39:56.190646+0000","last_became_peered":"2024-03-16T09:39:56.190646+0000","last_unstale":"2024-03-16T09:41:27.814777+0000","last_undegraded":"2024-03-16T09:41:27.814777+0000","last_fullsized":"2024-03-16T09:41:27.814777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:03.191129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-16T09:41:27.803969+0000","last_change":"2024-03-16T09:39:56.197552+0000","last_active":"2024-03-16T09:41:27.803969+0000","last_peered":"2024-03-16T09:41:27.803969+0000","last_clean":"2024-03-16T09:41:27.803969+0000","last_became_active":"2024-03-16T09:39:56.197215+0000","last_became_peered":"2024-03-16T09:39:56.197215+0000","last_unstale":"2024-03-16T09:41:27.803969+0000","last_undegraded":"2024-03-16T09:41:27.803969+0000","last_fullsized":"2024-03-16T09:41:27.803969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T09:39:55.160371+0000","last_clean_scrub_stamp":"2024-03-16T09:39:55.160371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:36:45.795328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64868,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":690,"num_read_kb":416,"num_write":169,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":460713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1113430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037812,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":460825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1053372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":460316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":520874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":461587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":522717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":461246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36876,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708276,"statfs":{"total":95995035648,"available":95957274624,"internally_reserved":0,"allocated":1576960,"data_stored":1052070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T09:41:53.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.799+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/142932639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc101330 msgr2=0x7f6ebc108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.799+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/142932639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc101330 0x7f6ebc108bd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009ca0 tx=0x7f6eb402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/142932639 shutdown_connections 2024-03-16T09:41:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/142932639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc101330 0x7f6ebc108bd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/142932639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ebc100970 0x7f6ebc100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/142932639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ebc101d20 0x7f6ebc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/142932639 >> 172.21.15.77:0/142932639 conn(0x7f6ebc0fb400 msgr2=0x7f6ebc0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/142932639 shutdown_connections 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/142932639 wait complete. 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 Processor -- start 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 -- start start 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc100970 0x7f6ebc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ebc101330 0x7f6ebc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.800+0000 7f6ec0cbf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ebc101d20 0x7f6ebc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6ec0cbf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc10c260 con 0x7f6ebc100970 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6ec0cbf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc10c0e0 con 0x7f6ebc101330 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6ec0cbf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc10c3e0 con 0x7f6ebc101d20 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc100970 0x7f6ebc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eb9d9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ebc101330 0x7f6ebc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc100970 0x7f6ebc197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37080/0 (socket says 172.21.15.77:37080) 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 -- 172.21.15.77:0/4125127513 learned_addr learned my addr 172.21.15.77:0/4125127513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6ebad9d640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ebc101d20 0x7f6ebc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 -- 172.21.15.77:0/4125127513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ebc101d20 msgr2=0x7f6ebc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ebc101d20 0x7f6ebc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 -- 172.21.15.77:0/4125127513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ebc101330 msgr2=0x7f6ebc197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ebc101330 0x7f6ebc197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 -- 172.21.15.77:0/4125127513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc19f670 con 0x7f6ebc100970 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eb9d9b640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ebc101330 0x7f6ebc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eba59c640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc100970 0x7f6ebc197a50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000e9c0 tx=0x7f6eb000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eab7fe640 1 -- 172.21.15.77:0/4125127513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb000cd80 con 0x7f6ebc100970 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ebc19f960 con 0x7f6ebc100970 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eab7fe640 1 -- 172.21.15.77:0/4125127513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eb00035f0 con 0x7f6ebc100970 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6eab7fe640 1 -- 172.21.15.77:0/4125127513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0010690 con 0x7f6ebc100970 2024-03-16T09:41:53.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.801+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ebc19feb0 con 0x7f6ebc100970 2024-03-16T09:41:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.802+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ebc1a0140 con 0x7f6ebc100970 2024-03-16T09:41:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.803+0000 7f6eab7fe640 1 -- 172.21.15.77:0/4125127513 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6eb00037a0 con 0x7f6ebc100970 2024-03-16T09:41:53.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.804+0000 7f6eab7fe640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea40a2800 0x7f6ea40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.804+0000 7f6eb9d9b640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea40a2800 0x7f6ea40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.804+0000 7f6eab7fe640 1 -- 172.21.15.77:0/4125127513 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6eb00cede0 con 0x7f6ebc100970 2024-03-16T09:41:53.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.804+0000 7f6eb9d9b640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea40a2800 0x7f6ea40a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc198f60 tx=0x7f6eac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.808+0000 7f6eab7fe640 1 -- 172.21.15.77:0/4125127513 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6eb009be90 con 0x7f6ebc100970 2024-03-16T09:41:53.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.988+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ebc1a0310 con 0x7f6ebc100970 2024-03-16T09:41:53.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.990+0000 7f6eab7fe640 1 -- 172.21.15.77:0/4125127513 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f6eb009b790 con 0x7f6ebc100970 2024-03-16T09:41:53.993 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:53.993 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:53.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea40a2800 msgr2=0x7f6ea40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea40a2800 0x7f6ea40a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc198f60 tx=0x7f6eac005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:53.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc100970 msgr2=0x7f6ebc197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:53.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc100970 0x7f6ebc197a50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000e9c0 tx=0x7f6eb000ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:53.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 shutdown_connections 2024-03-16T09:41:53.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea40a2800 0x7f6ea40a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ebc101d20 0x7f6ebc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ebc101330 0x7f6ebc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 --2- 172.21.15.77:0/4125127513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ebc100970 0x7f6ebc197a50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:53.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 >> 172.21.15.77:0/4125127513 conn(0x7f6ebc0fb400 msgr2=0x7f6ebc0fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:53.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 shutdown_connections 2024-03-16T09:41:53.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:53.992+0000 7f6ec0cbf640 1 -- 172.21.15.77:0/4125127513 wait complete. 2024-03-16T09:41:54.006 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:41:54.007 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:41:54.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 -- 172.21.15.77:0/2523072954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1910102a90 msgr2=0x7f1910102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 --2- 172.21.15.77:0/2523072954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1910102a90 0x7f1910102f10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1904009930 tx=0x7f190402bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 -- 172.21.15.77:0/2523072954 shutdown_connections 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 --2- 172.21.15.77:0/2523072954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1910103450 0x7f1910109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 --2- 172.21.15.77:0/2523072954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1910102a90 0x7f1910102f10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 --2- 172.21.15.77:0/2523072954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19100ffac0 0x7f19100ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 -- 172.21.15.77:0/2523072954 >> 172.21.15.77:0/2523072954 conn(0x7f19100fb410 msgr2=0x7f19100fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 -- 172.21.15.77:0/2523072954 shutdown_connections 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.109+0000 7f1914a97640 1 -- 172.21.15.77:0/2523072954 wait complete. 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 Processor -- start 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 -- start start 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19100ffac0 0x7f1910197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1910102a90 0x7f1910197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1910103450 0x7f191019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f191010d330 con 0x7f19100ffac0 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f191010d1b0 con 0x7f1910103450 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f1914a97640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f191010d4b0 con 0x7f1910102a90 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19100ffac0 0x7f1910197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19100ffac0 0x7f1910197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37102/0 (socket says 172.21.15.77:37102) 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 -- 172.21.15.77:0/132989978 learned_addr learned my addr 172.21.15.77:0/132989978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190dd9b640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1910102a90 0x7f1910197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190ed9d640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1910103450 0x7f191019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 -- 172.21.15.77:0/132989978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1910102a90 msgr2=0x7f1910197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1910102a90 0x7f1910197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 -- 172.21.15.77:0/132989978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1910103450 msgr2=0x7f191019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1910103450 0x7f191019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190e59c640 1 -- 172.21.15.77:0/132989978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191019f650 con 0x7f19100ffac0 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.110+0000 7f190ed9d640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1910103450 0x7f191019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.111+0000 7f190e59c640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19100ffac0 0x7f1910197a30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f190000e9c0 tx=0x7f190000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:54.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.111+0000 7f18ff7fe640 1 -- 172.21.15.77:0/132989978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190000cd80 con 0x7f19100ffac0 2024-03-16T09:41:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.111+0000 7f18ff7fe640 1 -- 172.21.15.77:0/132989978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19000035f0 con 0x7f19100ffac0 2024-03-16T09:41:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.111+0000 7f18ff7fe640 1 -- 172.21.15.77:0/132989978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900010690 con 0x7f19100ffac0 2024-03-16T09:41:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.111+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f191019f940 con 0x7f19100ffac0 2024-03-16T09:41:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.111+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f191019fe60 con 0x7f19100ffac0 2024-03-16T09:41:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.112+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1910068a20 con 0x7f19100ffac0 2024-03-16T09:41:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.113+0000 7f18ff7fe640 1 -- 172.21.15.77:0/132989978 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f19000037a0 con 0x7f19100ffac0 2024-03-16T09:41:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.113+0000 7f18ff7fe640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18f80a2800 0x7f18f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.113+0000 7f190dd9b640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18f80a2800 0x7f18f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.113+0000 7f18ff7fe640 1 -- 172.21.15.77:0/132989978 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f19000cee90 con 0x7f19100ffac0 2024-03-16T09:41:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.114+0000 7f190dd9b640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18f80a2800 0x7f18f80a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1904009900 tx=0x7f1904005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.118+0000 7f18ff7fe640 1 -- 172.21.15.77:0/132989978 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f190009bf40 con 0x7f19100ffac0 2024-03-16T09:41:54.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.293+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1910062480 con 0x7f19100ffac0 2024-03-16T09:41:54.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.293+0000 7f18ff7fe640 1 -- 172.21.15.77:0/132989978 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f190009b840 con 0x7f19100ffac0 2024-03-16T09:41:54.296 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:41:54.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18f80a2800 msgr2=0x7f18f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18f80a2800 0x7f18f80a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1904009900 tx=0x7f1904005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19100ffac0 msgr2=0x7f1910197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19100ffac0 0x7f1910197a30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f190000e9c0 tx=0x7f190000ee80 comp rx=0 tx=0).stop 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 shutdown_connections 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f18f80a2800 0x7f18f80a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1910103450 0x7f191019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1910102a90 0x7f1910197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 --2- 172.21.15.77:0/132989978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19100ffac0 0x7f1910197a30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 >> 172.21.15.77:0/132989978 conn(0x7f19100fb410 msgr2=0x7f1910105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 shutdown_connections 2024-03-16T09:41:54.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.295+0000 7f1914a97640 1 -- 172.21.15.77:0/132989978 wait complete. 2024-03-16T09:41:54.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 -- 172.21.15.77:0/779966460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301017f0 msgr2=0x7fdc30101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 --2- 172.21.15.77:0/779966460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301017f0 0x7fdc30101c10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009900 tx=0x7fdc2002bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 -- 172.21.15.77:0/779966460 shutdown_connections 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 --2- 172.21.15.77:0/779966460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc301033a0 0x7fdc30109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 --2- 172.21.15.77:0/779966460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc301029e0 0x7fdc30102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 --2- 172.21.15.77:0/779966460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301017f0 0x7fdc30101c10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 -- 172.21.15.77:0/779966460 >> 172.21.15.77:0/779966460 conn(0x7fdc300fcd80 msgr2=0x7fdc300ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.411+0000 7fdc366f8640 1 -- 172.21.15.77:0/779966460 shutdown_connections 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 -- 172.21.15.77:0/779966460 wait complete. 2024-03-16T09:41:54.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 Processor -- start 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 -- start start 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc301017f0 0x7fdc30197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc301029e0 0x7fdc30197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301033a0 0x7fdc3019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3010d310 con 0x7fdc301033a0 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3010d190 con 0x7fdc301017f0 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc366f8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3010d490 con 0x7fdc301029e0 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc34c95640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301033a0 0x7fdc3019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc34c95640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301033a0 0x7fdc3019f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37110/0 (socket says 172.21.15.77:37110) 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.412+0000 7fdc34c95640 1 -- 172.21.15.77:0/3771886802 learned_addr learned my addr 172.21.15.77:0/3771886802 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc2ffff640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc301017f0 0x7fdc30197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc34c95640 1 -- 172.21.15.77:0/3771886802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc301029e0 msgr2=0x7fdc30197fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T09:41:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc34c95640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc301029e0 0x7fdc30197fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc34c95640 1 -- 172.21.15.77:0/3771886802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc301017f0 msgr2=0x7fdc30197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc34c95640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc301017f0 0x7fdc30197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc34c95640 1 -- 172.21.15.77:0/3771886802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3019f690 con 0x7fdc301033a0 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc34c95640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301033a0 0x7fdc3019f020 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800eb10 tx=0x7fdc2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc2d7fa640 1 -- 172.21.15.77:0/3771886802 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28004b20 con 0x7fdc301033a0 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3019f980 con 0x7fdc301033a0 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc2d7fa640 1 -- 172.21.15.77:0/3771886802 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc28004cc0 con 0x7fdc301033a0 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc2d7fa640 1 -- 172.21.15.77:0/3771886802 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28010690 con 0x7fdc301033a0 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.413+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3019fed0 con 0x7fdc301033a0 2024-03-16T09:41:54.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.414+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc30108c50 con 0x7fdc301033a0 2024-03-16T09:41:54.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.416+0000 7fdc2d7fa640 1 -- 172.21.15.77:0/3771886802 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdc28005000 con 0x7fdc301033a0 2024-03-16T09:41:54.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.416+0000 7fdc2d7fa640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc180a2800 0x7fdc180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.416+0000 7fdc2ffff640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc180a2800 0x7fdc180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.416+0000 7fdc2d7fa640 1 -- 172.21.15.77:0/3771886802 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdc280cf290 con 0x7fdc301033a0 2024-03-16T09:41:54.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.417+0000 7fdc2ffff640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc180a2800 0x7fdc180a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005ab0 tx=0x7fdc20005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:54.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.420+0000 7fdc2d7fa640 1 -- 172.21.15.77:0/3771886802 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc28053740 con 0x7fdc301033a0 2024-03-16T09:41:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.594+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdc3002d360 con 0x7fdc301033a0 2024-03-16T09:41:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.594+0000 7fdc2d7fa640 1 -- 172.21.15.77:0/3771886802 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fdc2809c340 con 0x7fdc301033a0 2024-03-16T09:41:54.597 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:41:54.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc180a2800 msgr2=0x7fdc180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc180a2800 0x7fdc180a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005ab0 tx=0x7fdc20005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301033a0 msgr2=0x7fdc3019f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301033a0 0x7fdc3019f020 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800eb10 tx=0x7fdc2800c2d0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 shutdown_connections 2024-03-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdc180a2800 0x7fdc180a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc301033a0 0x7fdc3019f020 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc301029e0 0x7fdc30197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 --2- 172.21.15.77:0/3771886802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc301017f0 0x7fdc30197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 >> 172.21.15.77:0/3771886802 conn(0x7fdc300fcd80 msgr2=0x7fdc30105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.596+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 shutdown_connections 2024-03-16T09:41:54.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.597+0000 7fdc366f8640 1 -- 172.21.15.77:0/3771886802 wait complete. 2024-03-16T09:41:54.611 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:41:54.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/441907328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 msgr2=0x7f2fd4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/441907328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4101d70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009900 tx=0x7f2fd002bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/441907328 shutdown_connections 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/441907328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41022b0 0x7f2fd4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/441907328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4101d70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/441907328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fd4100700 0x7f2fd4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/441907328 >> 172.21.15.77:0/441907328 conn(0x7f2fd40fbc90 msgr2=0x7f2fd40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/441907328 shutdown_connections 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/441907328 wait complete. 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 Processor -- start 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- start start 2024-03-16T09:41:54.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd4100700 0x7f2fd4196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fd41022b0 0x7f2fd419ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410c210 con 0x7f2fd41018f0 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410c090 con 0x7f2fd4100700 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.713+0000 7f2fdc6fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410c390 con 0x7f2fd41022b0 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fda49a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd4100700 0x7f2fd4196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37130/0 (socket says 172.21.15.77:37130) 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 -- 172.21.15.77:0/3067642503 learned_addr learned my addr 172.21.15.77:0/3067642503 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 -- 172.21.15.77:0/3067642503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fd41022b0 msgr2=0x7f2fd419ded0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fd41022b0 0x7f2fd419ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 -- 172.21.15.77:0/3067642503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd4100700 msgr2=0x7f2fd4196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd4100700 0x7f2fd4196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 -- 172.21.15.77:0/3067642503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd419e540 con 0x7f2fd41018f0 2024-03-16T09:41:54.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fda49a640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd4100700 0x7f2fd4196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:54.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fd9c99640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4196e60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0006010 tx=0x7f2fd0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:54.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fcb7fe640 1 -- 172.21.15.77:0/3067642503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0043020 con 0x7f2fd41018f0 2024-03-16T09:41:54.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd419e7d0 con 0x7f2fd41018f0 2024-03-16T09:41:54.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fcb7fe640 1 -- 172.21.15.77:0/3067642503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fd00043d0 con 0x7f2fd41018f0 2024-03-16T09:41:54.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fcb7fe640 1 -- 172.21.15.77:0/3067642503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0003b80 con 0x7f2fd41018f0 2024-03-16T09:41:54.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.714+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd419ed20 con 0x7f2fd41018f0 2024-03-16T09:41:54.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.715+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd4107a70 con 0x7f2fd41018f0 2024-03-16T09:41:54.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.716+0000 7f2fcb7fe640 1 -- 172.21.15.77:0/3067642503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2fd0046020 con 0x7f2fd41018f0 2024-03-16T09:41:54.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.717+0000 7f2fcb7fe640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2fc00a2800 0x7f2fc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:54.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.717+0000 7f2fda49a640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2fc00a2800 0x7f2fc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:54.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.717+0000 7f2fcb7fe640 1 -- 172.21.15.77:0/3067642503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2fd00f1350 con 0x7f2fd41018f0 2024-03-16T09:41:54.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.718+0000 7f2fda49a640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2fc00a2800 0x7f2fc00a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4101750 tx=0x7f2fc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:54.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.721+0000 7f2fcb7fe640 1 -- 172.21.15.77:0/3067642503 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2fd004d020 con 0x7f2fd41018f0 2024-03-16T09:41:54.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.897+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2fd402d340 con 0x7f2fd41018f0 2024-03-16T09:41:54.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.897+0000 7f2fcb7fe640 1 -- 172.21.15.77:0/3067642503 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f2fd00be3a0 con 0x7f2fd41018f0 2024-03-16T09:41:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:41:54.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2fc00a2800 msgr2=0x7f2fc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2fc00a2800 0x7f2fc00a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4101750 tx=0x7f2fc4005f90 comp rx=0 tx=0).stop 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 msgr2=0x7f2fd4196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4196e60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0006010 tx=0x7f2fd0005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 shutdown_connections 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2fc00a2800 0x7f2fc00a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fd41022b0 0x7f2fd419ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fd41018f0 0x7f2fd4196e60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 --2- 172.21.15.77:0/3067642503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd4100700 0x7f2fd4196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.899+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 >> 172.21.15.77:0/3067642503 conn(0x7f2fd40fbc90 msgr2=0x7f2fd4104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.900+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 shutdown_connections 2024-03-16T09:41:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:54.900+0000 7f2fdc6fe640 1 -- 172.21.15.77:0/3067642503 wait complete. 2024-03-16T09:41:54.913 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:41:55.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.012+0000 7f71dfd87640 1 -- 172.21.15.77:0/3755292757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 msgr2=0x7f71d8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.012+0000 7f71dfd87640 1 --2- 172.21.15.77:0/3755292757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8101d90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009900 tx=0x7f71c802bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:55.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.012+0000 7f71dfd87640 1 -- 172.21.15.77:0/3755292757 shutdown_connections 2024-03-16T09:41:55.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.012+0000 7f71dfd87640 1 --2- 172.21.15.77:0/3755292757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d81022d0 0x7f71d8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.012+0000 7f71dfd87640 1 --2- 172.21.15.77:0/3755292757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8101d90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.012+0000 7f71dfd87640 1 --2- 172.21.15.77:0/3755292757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d8100720 0x7f71d8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.012+0000 7f71dfd87640 1 -- 172.21.15.77:0/3755292757 >> 172.21.15.77:0/3755292757 conn(0x7f71d80fbc90 msgr2=0x7f71d80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 -- 172.21.15.77:0/3755292757 shutdown_connections 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 -- 172.21.15.77:0/3755292757 wait complete. 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 Processor -- start 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 -- start start 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d8100720 0x7f71d8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d81022d0 0x7f71d819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f71d810c250 con 0x7f71d8101910 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f71d810c0d0 con 0x7f71d8100720 2024-03-16T09:41:55.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dfd87640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71d810c3d0 con 0x7f71d81022d0 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dd322640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dd322640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37138/0 (socket says 172.21.15.77:37138) 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71dd322640 1 -- 172.21.15.77:0/72169191 learned_addr learned my addr 172.21.15.77:0/72169191 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71ddb23640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d8100720 0x7f71d8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.013+0000 7f71de324640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d81022d0 0x7f71d819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dd322640 1 -- 172.21.15.77:0/72169191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d81022d0 msgr2=0x7f71d819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dd322640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d81022d0 0x7f71d819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dd322640 1 -- 172.21.15.77:0/72169191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d8100720 msgr2=0x7f71d8197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dd322640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d8100720 0x7f71d8197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dd322640 1 -- 172.21.15.77:0/72169191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d819f6a0 con 0x7f71d8101910 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dd322640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8197fc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f71c8006010 tx=0x7f71c8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71ceffd640 1 -- 172.21.15.77:0/72169191 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8043020 con 0x7f71d8101910 2024-03-16T09:41:55.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d819f930 con 0x7f71d8101910 2024-03-16T09:41:55.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71ceffd640 1 -- 172.21.15.77:0/72169191 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71c80043d0 con 0x7f71d8101910 2024-03-16T09:41:55.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71ceffd640 1 -- 172.21.15.77:0/72169191 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8003b80 con 0x7f71d8101910 2024-03-16T09:41:55.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.014+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d819fe80 con 0x7f71d8101910 2024-03-16T09:41:55.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.015+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d8068a20 con 0x7f71d8101910 2024-03-16T09:41:55.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.017+0000 7f71ceffd640 1 -- 172.21.15.77:0/72169191 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f71c8046020 con 0x7f71d8101910 2024-03-16T09:41:55.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.017+0000 7f71ceffd640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71c40a2800 0x7f71c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.017+0000 7f71ceffd640 1 -- 172.21.15.77:0/72169191 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f71c80f1260 con 0x7f71d8101910 2024-03-16T09:41:55.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.017+0000 7f71ddb23640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71c40a2800 0x7f71c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.018+0000 7f71ddb23640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71c40a2800 0x7f71c40a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f71d8101770 tx=0x7f71d4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.021+0000 7f71ceffd640 1 -- 172.21.15.77:0/72169191 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f71c804d020 con 0x7f71d8101910 2024-03-16T09:41:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:54 smithi119 ceph-mon[35632]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:54 smithi119 ceph-mon[35632]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:54 smithi119 ceph-mon[35632]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:54 smithi119 ceph-mon[35632]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:41:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:54 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/4125127513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:54 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/132989978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:54 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3771886802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:55.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.236+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71d81a02a0 con 0x7f71d8101910 2024-03-16T09:41:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.237+0000 7f71ceffd640 1 -- 172.21.15.77:0/72169191 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f71c80be310 con 0x7f71d8101910 2024-03-16T09:41:55.240 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:55.240 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:41:55.240 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:41:55.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71c40a2800 msgr2=0x7f71c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71c40a2800 0x7f71c40a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f71d8101770 tx=0x7f71d4009500 comp rx=0 tx=0).stop 2024-03-16T09:41:55.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 msgr2=0x7f71d8197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8197fc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f71c8006010 tx=0x7f71c8005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 shutdown_connections 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f71c40a2800 0x7f71c40a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d81022d0 0x7f71d819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71d8101910 0x7f71d8197fc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 --2- 172.21.15.77:0/72169191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d8100720 0x7f71d8197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 >> 172.21.15.77:0/72169191 conn(0x7f71d80fbc90 msgr2=0x7f71d80fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 shutdown_connections 2024-03-16T09:41:55.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.239+0000 7f71dfd87640 1 -- 172.21.15.77:0/72169191 wait complete. 2024-03-16T09:41:55.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:54 smithi077 ceph-mon[32171]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:54 smithi077 ceph-mon[32171]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:54 smithi077 ceph-mon[32171]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:54 smithi077 ceph-mon[32171]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:41:55.253 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:54 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/4125127513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:55.253 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:54 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/132989978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:55.253 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:54 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3771886802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:55.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:41:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:54 smithi129 ceph-mon[35841]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:54 smithi129 ceph-mon[35841]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:54 smithi129 ceph-mon[35841]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T09:41:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:54 smithi129 ceph-mon[35841]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:41:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:54 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/4125127513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:54 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/132989978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:41:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:54 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3771886802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:41:55.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.354+0000 7fe138219640 1 -- 172.21.15.77:0/1694202766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe130100700 msgr2=0x7fe130100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.354+0000 7fe138219640 1 --2- 172.21.15.77:0/1694202766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe130100700 0x7fe130100b20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe120009900 tx=0x7fe12002bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:55.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.354+0000 7fe138219640 1 -- 172.21.15.77:0/1694202766 shutdown_connections 2024-03-16T09:41:55.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.354+0000 7fe138219640 1 --2- 172.21.15.77:0/1694202766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1301022b0 0x7fe130108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.354+0000 7fe138219640 1 --2- 172.21.15.77:0/1694202766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1301018f0 0x7fe130101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.354+0000 7fe138219640 1 --2- 172.21.15.77:0/1694202766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe130100700 0x7fe130100b20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.354+0000 7fe138219640 1 -- 172.21.15.77:0/1694202766 >> 172.21.15.77:0/1694202766 conn(0x7fe1300fbc70 msgr2=0x7fe1300fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 -- 172.21.15.77:0/1694202766 shutdown_connections 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 -- 172.21.15.77:0/1694202766 wait complete. 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 Processor -- start 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 -- start start 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe130100700 0x7fe1300694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1301018f0 0x7fe13019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1301022b0 0x7fe13019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe13010c230 con 0x7fe1301018f0 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe13010c0b0 con 0x7fe130100700 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe138219640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe13010c3b0 con 0x7fe1301022b0 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe1357b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1301018f0 0x7fe13019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe1357b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1301018f0 0x7fe13019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37160/0 (socket says 172.21.15.77:37160) 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe1357b4640 1 -- 172.21.15.77:0/3292235568 learned_addr learned my addr 172.21.15.77:0/3292235568 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe1367b6640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1301022b0 0x7fe13019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.355+0000 7fe135fb5640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe130100700 0x7fe1300694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe1357b4640 1 -- 172.21.15.77:0/3292235568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1301022b0 msgr2=0x7fe13019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe1357b4640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1301022b0 0x7fe13019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe1357b4640 1 -- 172.21.15.77:0/3292235568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe130100700 msgr2=0x7fe1300694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe1357b4640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe130100700 0x7fe1300694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe1357b4640 1 -- 172.21.15.77:0/3292235568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13019f690 con 0x7fe1301018f0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe1357b4640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1301018f0 0x7fe13019c790 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00e9f0 tx=0x7fe12c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe126ffd640 1 -- 172.21.15.77:0/3292235568 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c00cd80 con 0x7fe1301018f0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13019f980 con 0x7fe1301018f0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe126ffd640 1 -- 172.21.15.77:0/3292235568 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe12c0108e0 con 0x7fe1301018f0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe126ffd640 1 -- 172.21.15.77:0/3292235568 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c018920 con 0x7fe1301018f0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.356+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13019ff20 con 0x7fe1301018f0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.357+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe130107af0 con 0x7fe1301018f0 2024-03-16T09:41:55.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.358+0000 7fe126ffd640 1 -- 172.21.15.77:0/3292235568 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe12c010a90 con 0x7fe1301018f0 2024-03-16T09:41:55.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.359+0000 7fe126ffd640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe11c0a2800 0x7fe11c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.359+0000 7fe135fb5640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe11c0a2800 0x7fe11c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.359+0000 7fe126ffd640 1 -- 172.21.15.77:0/3292235568 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe12c0ced40 con 0x7fe1301018f0 2024-03-16T09:41:55.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.359+0000 7fe135fb5640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe11c0a2800 0x7fe11c0a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe120005ab0 tx=0x7fe120005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.363+0000 7fe126ffd640 1 -- 172.21.15.77:0/3292235568 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe12c09b7a0 con 0x7fe1301018f0 2024-03-16T09:41:55.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.539+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1300623e0 con 0x7fe1301018f0 2024-03-16T09:41:55.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.541+0000 7fe126ffd640 1 -- 172.21.15.77:0/3292235568 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7fe12c09f840 con 0x7fe1301018f0 2024-03-16T09:41:55.544 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:41:55.544 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe11c0a2800 msgr2=0x7fe11c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe11c0a2800 0x7fe11c0a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe120005ab0 tx=0x7fe120005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1301018f0 msgr2=0x7fe13019c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1301018f0 0x7fe13019c790 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00e9f0 tx=0x7fe12c00eeb0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 shutdown_connections 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe11c0a2800 0x7fe11c0a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1301022b0 0x7fe13019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1301018f0 0x7fe13019c790 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 --2- 172.21.15.77:0/3292235568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe130100700 0x7fe1300694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 >> 172.21.15.77:0/3292235568 conn(0x7fe1300fbc70 msgr2=0x7fe130104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 shutdown_connections 2024-03-16T09:41:55.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.543+0000 7fe138219640 1 -- 172.21.15.77:0/3292235568 wait complete. 2024-03-16T09:41:55.557 DEBUG:tasks.fwd_scrub:Yielding 2024-03-16T09:41:55.558 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-16T09:41:55.569 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:41:55.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-16T09:41:55.569 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-16T09:41:55.569 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T09:41:55.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:41:55.570 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T09:41:55.572 INFO:teuthology.run_tasks:Running task workunit... 2024-03-16T09:41:55.584 INFO:tasks.workunit:Pulling workunits from ref 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:41:55.584 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-16T09:41:55.584 INFO:tasks.workunit:timeout=3h 2024-03-16T09:41:55.584 INFO:tasks.workunit:cleanup=True 2024-03-16T09:41:55.585 DEBUG:teuthology.orchestra.run.smithi077:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:55.599 INFO:teuthology.orchestra.run.smithi077.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:55.599 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-16T09:41:55.599 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-16T09:41:55.599 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T09:41:55.600 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-16T09:41:55.600 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 09:40:00.414256522 +0000 2024-03-16T09:41:55.600 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 09:40:00.414256522 +0000 2024-03-16T09:41:55.600 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 09:41:48.929396398 +0000 2024-03-16T09:41:55.600 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-03-16T09:41:55.600 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-16T09:41:55.600 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-16T09:41:55.673 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:41:55.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 -- 172.21.15.77:0/3230579715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded8100700 msgr2=0x7fded8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 --2- 172.21.15.77:0/3230579715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded8100700 0x7fded8100b20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009900 tx=0x7fdecc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:55.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 -- 172.21.15.77:0/3230579715 shutdown_connections 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 --2- 172.21.15.77:0/3230579715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fded81022b0 0x7fded8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 --2- 172.21.15.77:0/3230579715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded81018f0 0x7fded8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 --2- 172.21.15.77:0/3230579715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded8100700 0x7fded8100b20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 -- 172.21.15.77:0/3230579715 >> 172.21.15.77:0/3230579715 conn(0x7fded80fbc90 msgr2=0x7fded80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 -- 172.21.15.77:0/3230579715 shutdown_connections 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.676+0000 7fdede0c6640 1 -- 172.21.15.77:0/3230579715 wait complete. 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 Processor -- start 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 -- start start 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded8100700 0x7fded8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded81018f0 0x7fded8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fded81022b0 0x7fded819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fded810c230 con 0x7fded81018f0 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fded810c0b0 con 0x7fded8100700 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fdede0c6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fded810c3b0 con 0x7fded81022b0 2024-03-16T09:41:55.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded81018f0 0x7fded8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded81018f0 0x7fded8197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45668/0 (socket says 172.21.15.77:45668) 2024-03-16T09:41:55.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 -- 172.21.15.77:0/2973128926 learned_addr learned my addr 172.21.15.77:0/2973128926 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded77fe640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded8100700 0x7fded8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded7fff640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fded81022b0 0x7fded819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 -- 172.21.15.77:0/2973128926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fded81022b0 msgr2=0x7fded819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fded81022b0 0x7fded819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 -- 172.21.15.77:0/2973128926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded8100700 msgr2=0x7fded8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded8100700 0x7fded8197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.677+0000 7fded6ffd640 1 -- 172.21.15.77:0/2973128926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded819f660 con 0x7fded81018f0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.678+0000 7fded7fff640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fded81022b0 0x7fded819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.678+0000 7fded6ffd640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded81018f0 0x7fded8197f80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdec800e9f0 tx=0x7fdec800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.678+0000 7fded4ff9640 1 -- 172.21.15.77:0/2973128926 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec800cd80 con 0x7fded81018f0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.678+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded819f950 con 0x7fded81018f0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.678+0000 7fded4ff9640 1 -- 172.21.15.77:0/2973128926 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdec80108e0 con 0x7fded81018f0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.678+0000 7fded4ff9640 1 -- 172.21.15.77:0/2973128926 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8018920 con 0x7fded81018f0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.678+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded819fef0 con 0x7fded81018f0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fded4ff9640 1 -- 172.21.15.77:0/2973128926 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdec8010a90 con 0x7fded81018f0 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 Processor -- start 2024-03-16T09:41:55.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 -- start start 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923437990 0x563923437db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x56392343ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56392343d4f0 0x56392343f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56392339c0d0 con 0x563923438380 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x563923344c70 con 0x56392343d4f0 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.680+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563923398170 con 0x563923437990 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x56392343ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923437990 0x563923437db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56392343d4f0 0x56392343f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923437990 0x563923437db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:56494/0 (socket says 172.21.15.77:56494) 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 -- 172.21.15.77:0/1728411709 learned_addr learned my addr 172.21.15.77:0/1728411709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded4ff9640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdec00a2800 0x7fdec00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 -- 172.21.15.77:0/1728411709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56392343d4f0 msgr2=0x56392343f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 --2- 172.21.15.77:0/1728411709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56392343d4f0 0x56392343f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 -- 172.21.15.77:0/1728411709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 msgr2=0x56392343ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 --2- 172.21.15.77:0/1728411709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x56392343ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 -- 172.21.15.77:0/1728411709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639231787e0 con 0x563923437990 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded4ff9640 1 -- 172.21.15.77:0/2973128926 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdec80cee70 con 0x7fded81018f0 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdec00a2800 0x7fdec00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 --2- 172.21.15.77:0/1728411709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923437990 0x563923437db0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fded000f4b0 tx=0x7fded000f970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c8518fd9a44fb65 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fdebdffb640 1 -- 172.21.15.77:0/1728411709 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0018070 con 0x563923437990 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fdebdffb640 1 -- 172.21.15.77:0/1728411709 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fded0021070 con 0x563923437990 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fded7fff640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdec00a2800 0x7fdec00a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fded0020000 tx=0x7fded001e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.681+0000 7fdebdffb640 1 -- 172.21.15.77:0/1728411709 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0010990 con 0x563923437990 2024-03-16T09:41:55.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1728411709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923437990 msgr2=0x563923437db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1728411709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923437990 0x563923437db0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fded000f4b0 tx=0x7fded000f970 comp rx=0 tx=0).stop 2024-03-16T09:41:55.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1728411709 shutdown_connections 2024-03-16T09:41:55.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1728411709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56392343d4f0 0x56392343f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1728411709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x56392343ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1728411709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923437990 0x563923437db0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1728411709 >> 172.21.15.77:0/1728411709 conn(0x5639233ffc50 msgr2=0x563923433300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1728411709 shutdown_connections 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.682+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1728411709 wait complete. 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 -- start start 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 Processor -- start 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923441e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923438380 0x563923442390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56392343d4f0 0x5639234493d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639234404a0 con 0x56392343d4f0 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5639233580d0 con 0x563923437990 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563923440620 con 0x563923438380 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fdee0d8f740 10 client.0 _ll_register_callbacks cb 0x563923321760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fded7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56392343d4f0 0x5639234493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fded7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56392343d4f0 0x5639234493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45694/0 (socket says 172.21.15.77:45694) 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fded6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923438380 0x563923442390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fded77fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923441e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fded7fff640 1 -- 172.21.15.77:0/490187590 learned_addr learned my addr 172.21.15.77:0/490187590 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.683+0000 7fded7fff640 1 -- 172.21.15.77:0/490187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923438380 msgr2=0x563923442390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fded7fff640 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923438380 0x563923442390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fded7fff640 1 -- 172.21.15.77:0/490187590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 msgr2=0x563923441e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fded7fff640 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923441e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fded7fff640 1 -- 172.21.15.77:0/490187590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639231d8f70 con 0x56392343d4f0 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fded6ffd640 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923438380 0x563923442390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fded7fff640 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56392343d4f0 0x5639234493d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fded0009540 tx=0x7fded0010e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8cbdebd9e011cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdeb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/490187590 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0032060 con 0x56392343d4f0 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/490187590 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fded0003970 con 0x56392343d4f0 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fded0021070 con 0x56392343d4f0 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/490187590 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fded0018040 con 0x56392343d4f0 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdee0d8f740 10 client.15381 fetch_fsmap learned FSMap version 21 2024-03-16T09:41:55.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563923070170 con 0x56392343d4f0 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/490187590 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fded0021070 con 0x56392343d4f0 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/490187590 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fded0005230 con 0x56392343d4f0 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdee0d8f740 10 client.15381 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.684+0000 7fdee0d8f740 10 client.15381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.685+0000 7fdee0d8f740 20 client.15381 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.685+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56392344d820 0x56392344fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.685+0000 7fdee0d8f740 4 client.15381 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.685+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639233a9610 con 0x56392344d820 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.685+0000 7fded77fe640 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56392344d820 0x56392344fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.685+0000 7fded77fe640 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56392344d820 0x56392344fc40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=895374bfe6e057e8 server_cookie=5eaf9f803d64445e in_seq=0 out_seq=0 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.685+0000 7fdeb6ffd640 10 client.15381 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/490187590 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5639233a9610 con 0x56392344d820 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdeb6ffd640 10 client.15381 handle_command_reply: tid=0 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 client.15381 shutdown 2024-03-16T09:41:55.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56392343d4f0 msgr2=0x5639234493d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56392343d4f0 0x5639234493d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fded0009540 tx=0x7fded0010e80 comp rx=0 tx=0).stop 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 shutdown_connections 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56392344d820 0x56392344fc40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56392343d4f0 0x5639234493d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563923438380 0x563923442390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdeb6ffd640 0 client.15381 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/490187590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923441e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 >> 172.21.15.77:0/490187590 conn(0x5639233ffc50 msgr2=0x5639234324e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 shutdown_connections 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 1 -- 172.21.15.77:0/490187590 wait complete. 2024-03-16T09:41:55.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.686+0000 7fdee0d8f740 20 client.15381 trim_cache size 0 max 16384 2024-03-16T09:41:55.695 INFO:tasks.workunit.client.0.smithi077.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 Processor -- start 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 -- start start 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923437990 0x563923441850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923438380 0x563923449390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x563923455510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fded81ab470 con 0x563923437990 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fded81a8d40 con 0x563923438380 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fded81a6b90 con 0x56392343d4f0 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923437990 0x563923441850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923438380 0x563923449390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x563923455510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923437990 0x563923441850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45706/0 (socket says 172.21.15.77:45706) 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 -- 172.21.15.77:0/118299852 learned_addr learned my addr 172.21.15.77:0/118299852 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 -- 172.21.15.77:0/118299852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 msgr2=0x563923455510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x563923455510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 -- 172.21.15.77:0/118299852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923438380 msgr2=0x563923449390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923438380 0x563923449390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.704+0000 7fded77fe640 1 -- 172.21.15.77:0/118299852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563923438760 con 0x563923437990 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.705+0000 7fded7fff640 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x563923455510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.705+0000 7fded6ffd640 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923438380 0x563923449390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.705+0000 7fded77fe640 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923437990 0x563923441850 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdecc005ab0 tx=0x7fdecc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0405f31235c2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.705+0000 7fdebd7fa640 1 -- 172.21.15.77:0/118299852 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc04f020 con 0x563923437990 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.705+0000 7fdebd7fa640 1 -- 172.21.15.77:0/118299852 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdecc02cb80 con 0x563923437990 2024-03-16T09:41:55.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.705+0000 7fdebd7fa640 1 -- 172.21.15.77:0/118299852 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc049070 con 0x563923437990 2024-03-16T09:41:55.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- 172.21.15.77:0/118299852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923437990 msgr2=0x563923441850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923437990 0x563923441850 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdecc005ab0 tx=0x7fdecc005fd0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- 172.21.15.77:0/118299852 shutdown_connections 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x563923455510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923438380 0x563923449390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/118299852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923437990 0x563923441850 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- 172.21.15.77:0/118299852 >> 172.21.15.77:0/118299852 conn(0x5639233ffc50 msgr2=0x563923441140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- 172.21.15.77:0/118299852 shutdown_connections 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- 172.21.15.77:0/118299852 wait complete. 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- start start 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 Processor -- start 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923447310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x563923447850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x5639234487f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fded81a7200 con 0x563923438380 2024-03-16T09:41:55.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fded81a8670 con 0x563923437990 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fded81a78d0 con 0x56392343d4f0 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.706+0000 7fdee0d8f740 10 client.0 _ll_register_callbacks cb 0x7fded00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x563923447850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x563923447850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45714/0 (socket says 172.21.15.77:45714) 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 -- 172.21.15.77:0/1444123258 learned_addr learned my addr 172.21.15.77:0/1444123258 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded7fff640 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x5639234487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded77fe640 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923447310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 -- 172.21.15.77:0/1444123258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 msgr2=0x5639234487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x5639234487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 -- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 msgr2=0x563923447310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923447310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 -- 172.21.15.77:0/1444123258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563923448de0 con 0x563923438380 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fded6ffd640 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x563923447850 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdec80cc920 tx=0x7fdec8097b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9060b2ff37b8671 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdeb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:55.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/1444123258 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec809b7c0 con 0x563923438380 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/1444123258 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdec80cc0f0 con 0x563923438380 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdecc02cb80 con 0x563923438380 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/1444123258 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdec809f910 con 0x563923438380 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdee0d8f740 10 client.15387 fetch_fsmap learned FSMap version 21 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5639234b4360 con 0x563923438380 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.707+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/1444123258 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdecc02cb80 con 0x563923438380 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/1444123258 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fdec80afba0 con 0x563923438380 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdee0d8f740 10 client.15387 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdee0d8f740 10 client.15387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdee0d8f740 20 client.15387 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5639234b4d10 0x5639234b7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdee0d8f740 4 client.15387 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5639233a3960 con 0x5639234b4d10 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fded77fe640 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5639234b4d10 0x5639234b7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fdeb6ffd640 10 client.15387 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.708+0000 7fded77fe640 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5639234b4d10 0x5639234b7130 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35c9b90d460220e0 server_cookie=4a1dd7cab2101e4 in_seq=0 out_seq=0 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.709+0000 7fdeb6ffd640 1 -- 172.21.15.77:0/1444123258 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5639233a3960 con 0x5639234b4d10 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.709+0000 7fdeb6ffd640 10 client.15387 handle_command_reply: tid=0 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.709+0000 7fdee0d8f740 1 client.15387 shutdown 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.709+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 msgr2=0x563923447850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.709+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x563923447850 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdec80cc920 tx=0x7fdec8097b90 comp rx=0 tx=0).stop 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 shutdown_connections 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5639234b4d10 0x5639234b7130 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56392343d4f0 0x5639234487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563923438380 0x563923447850 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 --2- 172.21.15.77:0/1444123258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563923437990 0x563923447310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 >> 172.21.15.77:0/1444123258 conn(0x5639233ffc50 msgr2=0x563923436540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdeb6ffd640 0 client.15387 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 shutdown_connections 2024-03-16T09:41:55.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 1 -- 172.21.15.77:0/1444123258 wait complete. 2024-03-16T09:41:55.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdee0d8f740 20 client.15387 trim_cache size 0 max 16384 2024-03-16T09:41:55.718 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:55.718 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T09:41:55.718 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "8b2e5312-ebf6-4bf9-84fc-80d89b717b92", 2024-03-16T09:41:55.718 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T09:41:55.718 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdec00a2800 msgr2=0x7fdec00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdec00a2800 0x7fdec00a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fded0020000 tx=0x7fded001e000 comp rx=0 tx=0).stop 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded81018f0 msgr2=0x7fded8197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded81018f0 0x7fded8197f80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdec800e9f0 tx=0x7fdec800eeb0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 shutdown_connections 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdec00a2800 0x7fdec00a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fded81022b0 0x7fded819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded81018f0 0x7fded8197f80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 --2- 172.21.15.77:0/2973128926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded8100700 0x7fded8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 >> 172.21.15.77:0/2973128926 conn(0x7fded80fbc90 msgr2=0x7fded8069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 shutdown_connections 2024-03-16T09:41:55.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.710+0000 7fdede0c6640 1 -- 172.21.15.77:0/2973128926 wait complete. 2024-03-16T09:41:55.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:41:55.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:41:55.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 -- 172.21.15.77:0/943469993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 msgr2=0x7f846c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 --2- 172.21.15.77:0/943469993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c101d50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8460009900 tx=0x7f846002bc30 comp rx=0 tx=0).stop 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 -- 172.21.15.77:0/943469993 shutdown_connections 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 --2- 172.21.15.77:0/943469993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f846c102290 0x7f846c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 --2- 172.21.15.77:0/943469993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c101d50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 --2- 172.21.15.77:0/943469993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846c1006e0 0x7f846c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 -- 172.21.15.77:0/943469993 >> 172.21.15.77:0/943469993 conn(0x7f846c0fbc70 msgr2=0x7f846c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 -- 172.21.15.77:0/943469993 shutdown_connections 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.877+0000 7f8471c30640 1 -- 172.21.15.77:0/943469993 wait complete. 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 Processor -- start 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 -- start start 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f846c1006e0 0x7f846c0694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c19c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846c102290 0x7f846c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10c210 con 0x7f846c1018d0 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10c090 con 0x7f846c102290 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f8471c30640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10c390 con 0x7f846c1006e0 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f846affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c19c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f846affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c19c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45724/0 (socket says 172.21.15.77:45724) 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f846affd640 1 -- 172.21.15.77:0/1091949939 learned_addr learned my addr 172.21.15.77:0/1091949939 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f846bfff640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846c102290 0x7f846c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.878+0000 7f846b7fe640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f846c1006e0 0x7f846c0694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f846affd640 1 -- 172.21.15.77:0/1091949939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f846c1006e0 msgr2=0x7f846c0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f846affd640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f846c1006e0 0x7f846c0694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f846affd640 1 -- 172.21.15.77:0/1091949939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846c102290 msgr2=0x7f846c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f846affd640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846c102290 0x7f846c19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f846affd640 1 -- 172.21.15.77:0/1091949939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846c19f670 con 0x7f846c1018d0 2024-03-16T09:41:55.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f846bfff640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846c102290 0x7f846c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:55.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f846affd640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c19c770 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8460006010 tx=0x7f8460005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f8468ff9640 1 -- 172.21.15.77:0/1091949939 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8460043020 con 0x7f846c1018d0 2024-03-16T09:41:55.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f8468ff9640 1 -- 172.21.15.77:0/1091949939 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84600043d0 con 0x7f846c1018d0 2024-03-16T09:41:55.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f8468ff9640 1 -- 172.21.15.77:0/1091949939 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8460003b80 con 0x7f846c1018d0 2024-03-16T09:41:55.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f846c19f900 con 0x7f846c1018d0 2024-03-16T09:41:55.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.879+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f846c19fea0 con 0x7f846c1018d0 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 Processor -- start 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 -- start start 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83748590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da8374d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da83750160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da836ad500 con 0x55da83748b60 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55da83655be0 con 0x55da83748170 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f8474816740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da836a95a0 con 0x55da8374dcd0 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da8374d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da8374d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45728/0 (socket says 172.21.15.77:45728) 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 -- 172.21.15.77:0/2640314848 learned_addr learned my addr 172.21.15.77:0/2640314848 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846bfff640 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83748590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 -- 172.21.15.77:0/2640314848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 msgr2=0x55da83750160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T09:41:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da83750160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 -- 172.21.15.77:0/2640314848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 msgr2=0x55da83748590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83748590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 -- 172.21.15.77:0/2640314848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da836694b0 con 0x55da83748b60 2024-03-16T09:41:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846bfff640 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83748590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:55.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.882+0000 7f846b7fe640 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da8374d660 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f845c009ca0 tx=0x7f845c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4cb635c56abbad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.885+0000 7f8451ffb640 1 -- 172.21.15.77:0/2640314848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845c004220 con 0x55da83748b60 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.885+0000 7f8451ffb640 1 -- 172.21.15.77:0/2640314848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f845c0043c0 con 0x55da83748b60 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.885+0000 7f8451ffb640 1 -- 172.21.15.77:0/2640314848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845c019810 con 0x55da83748b60 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.886+0000 7f8474816740 1 -- 172.21.15.77:0/2640314848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 msgr2=0x55da8374d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.886+0000 7f8474816740 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da8374d660 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f845c009ca0 tx=0x7f845c011810 comp rx=0 tx=0).stop 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.886+0000 7f8474816740 1 -- 172.21.15.77:0/2640314848 shutdown_connections 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.886+0000 7f8474816740 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da83750160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.886+0000 7f8474816740 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da8374d660 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.886+0000 7f8474816740 1 --2- 172.21.15.77:0/2640314848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83748590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.886+0000 7f8474816740 1 -- 172.21.15.77:0/2640314848 >> 172.21.15.77:0/2640314848 conn(0x55da837436e0 msgr2=0x55da83743ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8468ff9640 1 -- 172.21.15.77:0/1091949939 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8460046020 con 0x7f846c1018d0 2024-03-16T09:41:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8474816740 1 -- 172.21.15.77:0/2640314848 shutdown_connections 2024-03-16T09:41:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8474816740 1 -- 172.21.15.77:0/2640314848 wait complete. 2024-03-16T09:41:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8474816740 1 -- start start 2024-03-16T09:41:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8474816740 1 Processor -- start 2024-03-16T09:41:55.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8468ff9640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f84540a2790 0x7f84540a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8474816740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83752650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.887+0000 7f8474816740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da83748b60 0x55da83752b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f8474816740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8374dcd0 0x55da83759bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f8474816740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da83750c80 con 0x55da8374dcd0 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f8474816740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55da83669160 con 0x55da83748170 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846bfff640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f84540a2790 0x7f84540a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f8474816740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da83750e00 con 0x55da83748b60 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f8468ff9640 1 -- 172.21.15.77:0/1091949939 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f84600f0ad0 con 0x7f846c1018d0 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f8474816740 10 client.0 _ll_register_callbacks cb 0x55da8364d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83752650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83752650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:54570/0 (socket says 172.21.15.77:54570) 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 -- 172.21.15.77:0/1220705776 learned_addr learned my addr 172.21.15.77:0/1220705776 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846affd640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da83748b60 0x55da83752b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846bfff640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8374dcd0 0x55da83759bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 -- 172.21.15.77:0/1220705776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da83748b60 msgr2=0x55da83752b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da83748b60 0x55da83752b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 -- 172.21.15.77:0/1220705776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8374dcd0 msgr2=0x55da83759bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8374dcd0 0x55da83759bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 -- 172.21.15.77:0/1220705776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da8348b7d0 con 0x55da83748170 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846bfff640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8374dcd0 0x55da83759bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846bfff640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f84540a2790 0x7f84540a4c50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55da837493c0 tx=0x7f846400f790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f846b7fe640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83752650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f845c009c70 tx=0x7f845c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bdadd09ac607187 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f844affd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f844affd640 1 -- 172.21.15.77:0/1220705776 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845c028070 con 0x55da83748170 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f844affd640 1 -- 172.21.15.77:0/1220705776 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f845c003900 con 0x55da83748170 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.888+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f845c0043c0 con 0x55da83748170 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f844affd640 1 -- 172.21.15.77:0/1220705776 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f845c034050 con 0x55da83748170 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 10 client.25289 fetch_fsmap learned FSMap version 21 2024-03-16T09:41:55.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55da833c13f0 con 0x55da83748170 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f844affd640 1 -- 172.21.15.77:0/1220705776 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f845c0043c0 con 0x55da83748170 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f844affd640 1 -- 172.21.15.77:0/1220705776 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f845c0049a0 con 0x55da83748170 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 10 client.25289 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 10 client.25289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 20 client.25289 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da8375dec0 0x55da837602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 4 client.25289 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da836baa40 con 0x55da8375dec0 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f846affd640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da8375dec0 0x55da837602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f846affd640 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da8375dec0 0x55da837602b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1e6d1a66a0445d6 server_cookie=2aeb9ebdfa57cc3f in_seq=0 out_seq=0 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.889+0000 7f844affd640 10 client.25289 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f844affd640 1 -- 172.21.15.77:0/1220705776 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da836baa40 con 0x55da8375dec0 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f844affd640 10 client.25289 handle_command_reply: tid=0 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 client.25289 shutdown 2024-03-16T09:41:55.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 msgr2=0x55da83752650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83752650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f845c009c70 tx=0x7f845c005dc0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 shutdown_connections 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da8375dec0 0x55da837602b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da8374dcd0 0x55da83759bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f844affd640 0 client.25289 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da83748b60 0x55da83752b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 --2- 172.21.15.77:0/1220705776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da83752650 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 >> 172.21.15.77:0/1220705776 conn(0x55da837436e0 msgr2=0x55da83742900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 shutdown_connections 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.890+0000 7f8474816740 1 -- 172.21.15.77:0/1220705776 wait complete. 2024-03-16T09:41:55.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.891+0000 7f8474816740 20 client.25289 trim_cache size 0 max 16384 2024-03-16T09:41:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 Processor -- start 2024-03-16T09:41:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 -- start start 2024-03-16T09:41:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da837511d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da83759b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da83751c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f846c1ab4f0 con 0x55da83748b60 2024-03-16T09:41:55.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f846c1a8dc0 con 0x55da83748170 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f8474816740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f846c1a6c10 con 0x55da8374dcd0 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.909+0000 7f846affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da83759b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da837511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da83751c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da83759b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45746/0 (socket says 172.21.15.77:45746) 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 -- 172.21.15.77:0/1435823496 learned_addr learned my addr 172.21.15.77:0/1435823496 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 -- 172.21.15.77:0/1435823496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 msgr2=0x55da83751c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da83751c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 -- 172.21.15.77:0/1435823496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 msgr2=0x55da837511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da837511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 -- 172.21.15.77:0/1435823496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da83760620 con 0x55da83748b60 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846b7fe640 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da837511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f846affd640 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da83759b90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8460005c20 tx=0x7f84600dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85b6fc474a2962f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f84517fa640 1 -- 172.21.15.77:0/1435823496 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84600d5070 con 0x55da83748b60 2024-03-16T09:41:55.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f84517fa640 1 -- 172.21.15.77:0/1435823496 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84600bdb80 con 0x55da83748b60 2024-03-16T09:41:55.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.910+0000 7f84517fa640 1 -- 172.21.15.77:0/1435823496 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84600bb780 con 0x55da83748b60 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 -- 172.21.15.77:0/1435823496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 msgr2=0x55da83759b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da83759b90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8460005c20 tx=0x7f84600dd040 comp rx=0 tx=0).stop 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 -- 172.21.15.77:0/1435823496 shutdown_connections 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da83751c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748b60 0x55da83759b90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 --2- 172.21.15.77:0/1435823496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748170 0x55da837511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 -- 172.21.15.77:0/1435823496 >> 172.21.15.77:0/1435823496 conn(0x55da837436e0 msgr2=0x55da83742900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 -- 172.21.15.77:0/1435823496 shutdown_connections 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 -- 172.21.15.77:0/1435823496 wait complete. 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 -- start start 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.911+0000 7f8474816740 1 Processor -- start 2024-03-16T09:41:55.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748170 0x55da83757070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748b60 0x55da837575b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da837585c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f846c1a7280 con 0x55da83748170 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f846c1a86f0 con 0x55da83748b60 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f846c1a7950 con 0x55da8374dcd0 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748170 0x55da83757070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 10 client.0 _ll_register_callbacks cb 0x7f8464009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748170 0x55da83757070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45750/0 (socket says 172.21.15.77:45750) 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 -- 172.21.15.77:0/2192670465 learned_addr learned my addr 172.21.15.77:0/2192670465 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846affd640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748b60 0x55da837575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846bfff640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da837585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 -- 172.21.15.77:0/2192670465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 msgr2=0x55da837585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da837585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 -- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748b60 msgr2=0x55da837575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748b60 0x55da837575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 -- 172.21.15.77:0/2192670465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da833810f0 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846bfff640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da837585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f846b7fe640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748170 0x55da83757070 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f845c009230 tx=0x7f845c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9ea57088860d52 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f844affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f844affd640 1 -- 172.21.15.77:0/2192670465 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845c028040 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f844affd640 1 -- 172.21.15.77:0/2192670465 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f845c019640 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.912+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84600bdb80 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f844affd640 1 -- 172.21.15.77:0/2192670465 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f845c003bb0 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 10 client.15405 fetch_fsmap learned FSMap version 21 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55da836749d0 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f844affd640 1 -- 172.21.15.77:0/2192670465 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84600bdb80 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f844affd640 1 -- 172.21.15.77:0/2192670465 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f845c005140 con 0x55da83748170 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 10 client.15405 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 10 client.15405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 20 client.15405 populate_metadata read hostname 'smithi077' 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da837c4520 0x55da837c6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:41:55.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 4 client.15405 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da836b4d90 con 0x55da837c4520 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f846affd640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da837c4520 0x55da837c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f844affd640 10 client.15405 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.913+0000 7f846affd640 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da837c4520 0x55da837c6940 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9630161f853ae934 server_cookie=eb5931c1ac4229fa in_seq=0 out_seq=0 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f844affd640 1 -- 172.21.15.77:0/2192670465 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55da836b4d90 con 0x55da837c4520 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f844affd640 10 client.15405 handle_command_reply: tid=0 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 client.15405 shutdown 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748170 msgr2=0x55da83757070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748170 0x55da83757070 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f845c009230 tx=0x7f845c0093f0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 shutdown_connections 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da837c4520 0x55da837c6940 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da8374dcd0 0x55da837585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f844affd640 0 client.15405 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da83748b60 0x55da837575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 --2- 172.21.15.77:0/2192670465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da83748170 0x55da83757070 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 >> 172.21.15.77:0/2192670465 conn(0x55da837436e0 msgr2=0x55da83742070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 shutdown_connections 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 1 -- 172.21.15.77:0/2192670465 wait complete. 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8474816740 20 client.15405 trim_cache size 0 max 16384 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stdout: "8b2e5312-ebf6-4bf9-84fc-80d89b717b92": { 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "8b2e5312-ebf6-4bf9-84fc-80d89b717b92", 2024-03-16T09:41:55.923 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T09:41:55.924 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:41:55.924 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:41:55.924 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f84540a2790 msgr2=0x7f84540a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f84540a2790 0x7f84540a4c50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55da837493c0 tx=0x7f846400f790 comp rx=0 tx=0).stop 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 msgr2=0x7f846c19c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c19c770 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8460006010 tx=0x7f8460005e40 comp rx=0 tx=0).stop 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 shutdown_connections 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f84540a2790 0x7f84540a4c50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846c102290 0x7f846c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f846c1018d0 0x7f846c19c770 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 --2- 172.21.15.77:0/1091949939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f846c1006e0 0x7f846c0694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.914+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 >> 172.21.15.77:0/1091949939 conn(0x7f846c0fbc70 msgr2=0x7f846c104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.915+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 shutdown_connections 2024-03-16T09:41:55.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:41:55.915+0000 7f8471c30640 1 -- 172.21.15.77:0/1091949939 wait complete. 2024-03-16T09:41:55.928 INFO:tasks.cephfs.filesystem:scrub status for tag:8b2e5312-ebf6-4bf9-84fc-80d89b717b92 - {'path': '/', 'tag': '8b2e5312-ebf6-4bf9-84fc-80d89b717b92', 'options': 'recursive,force'} 2024-03-16T09:41:56.089 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3067642503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:56.089 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/72169191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:56.089 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:55 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3292235568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:56.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3067642503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:56.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/72169191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:56.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:55 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3292235568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:56.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3067642503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:41:56.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/72169191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:41:56.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:55 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3292235568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:41:57.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:56 smithi119 ceph-mon[35632]: scrub queued for path: / 2024-03-16T09:41:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:56 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:41:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:56 smithi119 ceph-mon[35632]: scrub summary: active paths [/] 2024-03-16T09:41:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:56 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:41:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:56 smithi119 ceph-mon[35632]: pgmap v347: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-16T09:41:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:56 smithi077 ceph-mon[32171]: scrub queued for path: / 2024-03-16T09:41:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:56 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:41:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:56 smithi077 ceph-mon[32171]: scrub summary: active paths [/] 2024-03-16T09:41:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:56 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:41:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:56 smithi077 ceph-mon[32171]: pgmap v347: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-16T09:41:57.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:56 smithi129 ceph-mon[35841]: scrub queued for path: / 2024-03-16T09:41:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:56 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:41:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:56 smithi129 ceph-mon[35841]: scrub summary: active paths [/] 2024-03-16T09:41:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:56 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:41:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:56 smithi129 ceph-mon[35841]: pgmap v347: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-16T09:41:59.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:41:58 smithi119 ceph-mon[35632]: pgmap v348: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-16T09:41:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:41:58 smithi077 ceph-mon[32171]: pgmap v348: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-16T09:41:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:41:58 smithi129 ceph-mon[35841]: pgmap v348: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-16T09:42:01.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:00 smithi119 ceph-mon[35632]: pgmap v349: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T09:42:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:00 smithi119 ceph-mon[35632]: scrub summary: idle 2024-03-16T09:42:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:00 smithi077 ceph-mon[32171]: pgmap v349: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T09:42:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:00 smithi077 ceph-mon[32171]: scrub summary: idle 2024-03-16T09:42:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:00 smithi129 ceph-mon[35841]: pgmap v349: 97 pgs: 97 active+clean; 645 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T09:42:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:00 smithi129 ceph-mon[35841]: scrub summary: idle 2024-03-16T09:42:03.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:02 smithi119 ceph-mon[35632]: pgmap v350: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-16T09:42:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:02 smithi077 ceph-mon[32171]: pgmap v350: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-16T09:42:03.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:02 smithi129 ceph-mon[35841]: pgmap v350: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-16T09:42:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:04 smithi119 ceph-mon[35632]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T09:42:05.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:04 smithi077 ceph-mon[32171]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T09:42:05.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:04 smithi129 ceph-mon[35841]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-16T09:42:07.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:06 smithi119 ceph-mon[35632]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-16T09:42:07.261 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:06 smithi077 ceph-mon[32171]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-16T09:42:07.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:06 smithi129 ceph-mon[35841]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 -- 172.21.15.77:0/3314512973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 msgr2=0x7f1740108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 --2- 172.21.15.77:0/3314512973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f1740108b50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1738009ca0 tx=0x7f173802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 -- 172.21.15.77:0/3314512973 shutdown_connections 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 --2- 172.21.15.77:0/3314512973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f1740108b50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 --2- 172.21.15.77:0/3314512973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1740101900 0x7f1740101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 --2- 172.21.15.77:0/3314512973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1740100710 0x7f1740100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 -- 172.21.15.77:0/3314512973 >> 172.21.15.77:0/3314512973 conn(0x7f17400fbca0 msgr2=0x7f17400fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 -- 172.21.15.77:0/3314512973 shutdown_connections 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.707+0000 7f1744873640 1 -- 172.21.15.77:0/3314512973 wait complete. 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 Processor -- start 2024-03-16T09:42:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 -- start start 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1740100710 0x7f1740196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1740101900 0x7f1740196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f174019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f174010c240 con 0x7f17401022c0 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f174010c0c0 con 0x7f1740100710 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.708+0000 7f1744873640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f174010c3c0 con 0x7f1740101900 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f174019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1740100710 0x7f1740196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f174019ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57772/0 (socket says 172.21.15.77:57772) 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173ed9d640 1 -- 172.21.15.77:0/1800535963 learned_addr learned my addr 172.21.15.77:0/1800535963 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f1737fff640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1740101900 0x7f1740196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173e59c640 1 -- 172.21.15.77:0/1800535963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1740101900 msgr2=0x7f1740196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173e59c640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1740101900 0x7f1740196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173e59c640 1 -- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 msgr2=0x7f174019ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173e59c640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f174019ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173e59c640 1 -- 172.21.15.77:0/1800535963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174019e440 con 0x7f1740100710 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173ed9d640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f174019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f173e59c640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1740100710 0x7f1740196820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f173000ead0 tx=0x7f173000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f17377fe640 1 -- 172.21.15.77:0/1800535963 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173001c070 con 0x7f1740100710 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174006a0c0 con 0x7f1740100710 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f17377fe640 1 -- 172.21.15.77:0/1800535963 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f173000bb70 con 0x7f1740100710 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f17377fe640 1 -- 172.21.15.77:0/1800535963 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730003d10 con 0x7f1740100710 2024-03-16T09:42:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.709+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174006a610 con 0x7f1740100710 2024-03-16T09:42:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.711+0000 7f17377fe640 1 -- 172.21.15.77:0/1800535963 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f173001f070 con 0x7f1740100710 2024-03-16T09:42:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 Processor -- start 2024-03-16T09:42:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 -- start start 2024-03-16T09:42:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b0d30 0x55dec32b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f17377fe640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f172c0a28a0 0x7f172c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b5ea0 0x55dec32b8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dec3215a40 con 0x55dec32b5ea0 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55dec31be920 con 0x55dec32b0340 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f174745e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dec3211ae0 con 0x55dec32b0d30 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f173ed9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f1737fff640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f172c0a28a0 0x7f172c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f173ed9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32b0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:33000/0 (socket says 172.21.15.77:33000) 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f173ed9d640 1 -- 172.21.15.77:0/3984608664 learned_addr learned my addr 172.21.15.77:0/3984608664 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f17377fe640 1 -- 172.21.15.77:0/1800535963 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f17300d0100 con 0x7f1740100710 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.712+0000 7f173ed9d640 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b5ea0 0x55dec32b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f173ed9d640 1 -- 172.21.15.77:0/3984608664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b0d30 msgr2=0x55dec32b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f173ed9d640 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b0d30 0x55dec32b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f173ed9d640 1 -- 172.21.15.77:0/3984608664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b5ea0 msgr2=0x55dec32b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f173ed9d640 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b5ea0 0x55dec32b8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f173ed9d640 1 -- 172.21.15.77:0/3984608664 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec31d6460 con 0x55dec32b0340 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f1737fff640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f172c0a28a0 0x7f172c0a4d60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1740197d30 tx=0x7f1728005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f173ed9d640 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32b0760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1738037040 tx=0x7f173802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eb3ec7132023785 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f1734ff9640 1 -- 172.21.15.77:0/3984608664 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173804a070 con 0x55dec32b0340 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f1734ff9640 1 -- 172.21.15.77:0/3984608664 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1738003b70 con 0x55dec32b0340 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.713+0000 7f1734ff9640 1 -- 172.21.15.77:0/3984608664 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738039040 con 0x55dec32b0340 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 -- 172.21.15.77:0/3984608664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 msgr2=0x55dec32b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32b0760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1738037040 tx=0x7f173802e040 comp rx=0 tx=0).stop 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 -- 172.21.15.77:0/3984608664 shutdown_connections 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b5ea0 0x55dec32b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b0d30 0x55dec32b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 --2- 172.21.15.77:0/3984608664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32b0760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 -- 172.21.15.77:0/3984608664 >> 172.21.15.77:0/3984608664 conn(0x55dec32ab8b0 msgr2=0x55dec32abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 -- 172.21.15.77:0/3984608664 shutdown_connections 2024-03-16T09:42:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 -- 172.21.15.77:0/3984608664 wait complete. 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 -- start start 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.714+0000 7f174745e740 1 Processor -- start 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32ba700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32c1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dec32b8df0 con 0x55dec32b0340 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55dec31d1d70 con 0x55dec32b0d30 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dec32b8f70 con 0x55dec32b5ea0 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 10 client.0 _ll_register_callbacks cb 0x55dec2f29a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32ba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32ba700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57788/0 (socket says 172.21.15.77:57788) 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 -- 172.21.15.77:0/2252197240 learned_addr learned my addr 172.21.15.77:0/2252197240 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f1737fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173ed9d640 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32c1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 -- 172.21.15.77:0/2252197240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 msgr2=0x55dec32c1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32c1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 -- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 msgr2=0x55dec32bac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 -- 172.21.15.77:0/2252197240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec2f2a210 con 0x55dec32b0340 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f1737fff640 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f173e59c640 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32ba700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1730099530 tx=0x7f1730003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4005342b2d7b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f1725ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f1725ffb640 1 -- 172.21.15.77:0/2252197240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17300afdd0 con 0x55dec32b0340 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f1725ffb640 1 -- 172.21.15.77:0/2252197240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f17300aff70 con 0x55dec32b0340 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.715+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1738003b70 con 0x55dec32b0340 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f1725ffb640 1 -- 172.21.15.77:0/2252197240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f173009bfb0 con 0x55dec32b0340 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 10 client.15417 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dec2ff6620 con 0x55dec32b0340 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f1725ffb640 1 -- 172.21.15.77:0/2252197240 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1738003b70 con 0x55dec32b0340 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f1725ffb640 1 -- 172.21.15.77:0/2252197240 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f173009ba20 con 0x55dec32b0340 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 10 client.15417 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 10 client.15417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 20 client.15417 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec32c6510 0x55dec32c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 4 client.15417 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dec3222f80 con 0x55dec32c6510 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f1737fff640 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec32c6510 0x55dec32c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f1725ffb640 10 client.15417 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.716+0000 7f1737fff640 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec32c6510 0x55dec32c8900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d8bbf8ead6bfcb5 server_cookie=d322825330ad8e0d in_seq=0 out_seq=0 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f1725ffb640 1 -- 172.21.15.77:0/2252197240 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dec3222f80 con 0x55dec32c6510 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f1725ffb640 10 client.15417 handle_command_reply: tid=0 2024-03-16T09:42:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 client.15417 shutdown 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 msgr2=0x55dec32ba700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32ba700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1730099530 tx=0x7f1730003730 comp rx=0 tx=0).stop 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 shutdown_connections 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec32c6510 0x55dec32c8900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32c1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f1725ffb640 0 client.15417 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 --2- 172.21.15.77:0/2252197240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32ba700 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 >> 172.21.15.77:0/2252197240 conn(0x55dec32ab8b0 msgr2=0x55dec32aabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 shutdown_connections 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 1 -- 172.21.15.77:0/2252197240 wait complete. 2024-03-16T09:42:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.717+0000 7f174745e740 20 client.15417 trim_cache size 0 max 16384 2024-03-16T09:42:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 Processor -- start 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 -- start start 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32baa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0d30 0x55dec32c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32aa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17401a91b0 con 0x55dec32b0d30 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f17401a6a80 con 0x55dec32b0340 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.732+0000 7f174745e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17401a48d0 con 0x55dec32b5ea0 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0d30 0x55dec32c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0d30 0x55dec32c1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57804/0 (socket says 172.21.15.77:57804) 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 -- 172.21.15.77:0/3480880641 learned_addr learned my addr 172.21.15.77:0/3480880641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f173e59c640 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f173ed9d640 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 -- 172.21.15.77:0/3480880641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 msgr2=0x55dec32aa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32aa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 -- 172.21.15.77:0/3480880641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 msgr2=0x55dec32baa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32baa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 -- 172.21.15.77:0/3480880641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec32c8c70 con 0x55dec32b0d30 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f173ed9d640 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1737fff640 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0d30 0x55dec32c1c10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f17280118d0 tx=0x7f1728014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa3dd62dfab0d18 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1727fff640 1 -- 172.21.15.77:0/3480880641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728029250 con 0x55dec32b0d30 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1727fff640 1 -- 172.21.15.77:0/3480880641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f172801d4d0 con 0x55dec32b0d30 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.733+0000 7f1727fff640 1 -- 172.21.15.77:0/3480880641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728028650 con 0x55dec32b0d30 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- 172.21.15.77:0/3480880641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0d30 msgr2=0x55dec32c1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0d30 0x55dec32c1c10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f17280118d0 tx=0x7f1728014000 comp rx=0 tx=0).stop 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- 172.21.15.77:0/3480880641 shutdown_connections 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec32aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0d30 0x55dec32c1c10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 --2- 172.21.15.77:0/3480880641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0340 0x55dec32baa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- 172.21.15.77:0/3480880641 >> 172.21.15.77:0/3480880641 conn(0x55dec32ab8b0 msgr2=0x55dec32ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- 172.21.15.77:0/3480880641 shutdown_connections 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- 172.21.15.77:0/3480880641 wait complete. 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- start start 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 Processor -- start 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32bebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec332a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17401a4f40 con 0x55dec32b0340 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f17401a63b0 con 0x55dec32b0d30 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17401a5610 con 0x55dec32b5ea0 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f174745e740 10 client.0 _ll_register_callbacks cb 0x7f17300056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f173e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32bebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f173e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32bebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57808/0 (socket says 172.21.15.77:57808) 2024-03-16T09:42:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f173e59c640 1 -- 172.21.15.77:0/4147175231 learned_addr learned my addr 172.21.15.77:0/4147175231 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.734+0000 7f1737fff640 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f173ed9d640 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec332a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f173e59c640 1 -- 172.21.15.77:0/4147175231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 msgr2=0x55dec332a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f173e59c640 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec332a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f173e59c640 1 -- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 msgr2=0x55dec32bf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f173e59c640 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f173e59c640 1 -- 172.21.15.77:0/4147175231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec332add0 con 0x55dec32b0340 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f173e59c640 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32bebc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f173009c860 tx=0x7f17300a5380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b439f31dc21ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f1725ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f1725ffb640 1 -- 172.21.15.77:0/4147175231 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173009c4c0 con 0x55dec32b0340 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f1725ffb640 1 -- 172.21.15.77:0/4147175231 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f17300b5b70 con 0x55dec32b0340 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f172801d4d0 con 0x55dec32b0340 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f1725ffb640 1 -- 172.21.15.77:0/4147175231 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f17300bb390 con 0x55dec32b0340 2024-03-16T09:42:08.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f174745e740 10 client.15423 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dec332c1b0 con 0x55dec32b0340 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f1725ffb640 1 -- 172.21.15.77:0/4147175231 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f172801d4d0 con 0x55dec32b0340 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.735+0000 7f1725ffb640 1 -- 172.21.15.77:0/4147175231 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1730087170 con 0x55dec32b0340 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f174745e740 10 client.15423 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f174745e740 10 client.15423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f174745e740 20 client.15423 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f174745e740 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec332cbb0 0x55dec332efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f174745e740 4 client.15423 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dec321d2d0 con 0x55dec332cbb0 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f1737fff640 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec332cbb0 0x55dec332efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f1725ffb640 10 client.15423 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.736+0000 7f1737fff640 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec332cbb0 0x55dec332efa0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=768044c5e4636d server_cookie=b44476c8a495b58e in_seq=0 out_seq=0 2024-03-16T09:42:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f1725ffb640 1 -- 172.21.15.77:0/4147175231 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dec321d2d0 con 0x55dec332cbb0 2024-03-16T09:42:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f1725ffb640 10 client.15423 handle_command_reply: tid=0 2024-03-16T09:42:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 client.15423 shutdown 2024-03-16T09:42:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 msgr2=0x55dec32bebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32bebc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f173009c860 tx=0x7f17300a5380 comp rx=0 tx=0).stop 2024-03-16T09:42:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 shutdown_connections 2024-03-16T09:42:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55dec332cbb0 0x55dec332efa0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dec32b5ea0 0x55dec332a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dec32b0d30 0x55dec32bf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 --2- 172.21.15.77:0/4147175231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dec32b0340 0x55dec32bebc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f1725ffb640 0 client.15423 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 >> 172.21.15.77:0/4147175231 conn(0x55dec32ab8b0 msgr2=0x55dec32bab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 shutdown_connections 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 1 -- 172.21.15.77:0/4147175231 wait complete. 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.747+0000 7f174745e740 20 client.15423 trim_cache size 0 max 16384 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:42:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:42:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f172c0a28a0 msgr2=0x7f172c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f172c0a28a0 0x7f172c0a4d60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1740197d30 tx=0x7f1728005f90 comp rx=0 tx=0).stop 2024-03-16T09:42:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1740100710 msgr2=0x7f1740196820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1740100710 0x7f1740196820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f173000ead0 tx=0x7f173000ef90 comp rx=0 tx=0).stop 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 shutdown_connections 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f172c0a28a0 0x7f172c0a4d60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401022c0 0x7f174019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1740101900 0x7f1740196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 --2- 172.21.15.77:0/1800535963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1740100710 0x7f1740196820 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 >> 172.21.15.77:0/1800535963 conn(0x7f17400fbca0 msgr2=0x7f1740104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 shutdown_connections 2024-03-16T09:42:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:08.748+0000 7f1744873640 1 -- 172.21.15.77:0/1800535963 wait complete. 2024-03-16T09:42:09.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:08 smithi119 ceph-mon[35632]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:42:09.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:08 smithi077 ceph-mon[32171]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:42:09.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:08 smithi129 ceph-mon[35841]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:42:11.173 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:10 smithi119 ceph-mon[35632]: pgmap v354: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-16T09:42:11.173 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:10 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:42:11.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:10 smithi129 ceph-mon[35841]: pgmap v354: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-16T09:42:11.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:10 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:42:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:10 smithi077 ceph-mon[32171]: pgmap v354: 97 pgs: 97 active+clean; 650 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-16T09:42:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:10 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: pgmap v355: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:42:13.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: pgmap v355: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:42:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: pgmap v355: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:42:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:42:13.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:42:15.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:14 smithi119 ceph-mon[35632]: pgmap v356: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T09:42:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:14 smithi077 ceph-mon[32171]: pgmap v356: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T09:42:15.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:14 smithi129 ceph-mon[35841]: pgmap v356: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T09:42:17.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:16 smithi119 ceph-mon[35632]: pgmap v357: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-03-16T09:42:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:16 smithi077 ceph-mon[32171]: pgmap v357: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-03-16T09:42:17.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:16 smithi129 ceph-mon[35841]: pgmap v357: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-03-16T09:42:19.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:18 smithi119 ceph-mon[35632]: pgmap v358: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T09:42:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:18 smithi077 ceph-mon[32171]: pgmap v358: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T09:42:19.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:18 smithi129 ceph-mon[35841]: pgmap v358: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T09:42:21.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:20 smithi119 ceph-mon[35632]: pgmap v359: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-03-16T09:42:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:20 smithi077 ceph-mon[32171]: pgmap v359: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-03-16T09:42:21.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:20 smithi129 ceph-mon[35841]: pgmap v359: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 -- 172.21.15.77:0/596657729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848102340 msgr2=0x7fe848104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/596657729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848102340 0x7fe848104730 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe838009900 tx=0x7fe83802bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 -- 172.21.15.77:0/596657729 shutdown_connections 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/596657729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8481075a0 0x7fe848109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/596657729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe848104c70 0x7fe848107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/596657729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848102340 0x7fe848104730 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 -- 172.21.15.77:0/596657729 >> 172.21.15.77:0/596657729 conn(0x7fe8480fbcb0 msgr2=0x7fe8480fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.845+0000 7fe84e9c6640 1 -- 172.21.15.77:0/596657729 shutdown_connections 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.846+0000 7fe84e9c6640 1 -- 172.21.15.77:0/596657729 wait complete. 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.846+0000 7fe84e9c6640 1 Processor -- start 2024-03-16T09:42:21.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.846+0000 7fe84e9c6640 1 -- start start 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.846+0000 7fe84e9c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe848102340 0x7fe8481979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84e9c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848104c70 0x7fe848197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84e9c6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8481075a0 0x7fe84819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84e9c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe84810d360 con 0x7fe848104c70 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84e9c6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe84810d1e0 con 0x7fe848102340 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84e9c6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe84810d4e0 con 0x7fe8481075a0 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848104c70 0x7fe848197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe847fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe848102340 0x7fe8481979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848104c70 0x7fe848197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58560/0 (socket says 172.21.15.77:58560) 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 -- 172.21.15.77:0/107355235 learned_addr learned my addr 172.21.15.77:0/107355235 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84cf63640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8481075a0 0x7fe84819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 -- 172.21.15.77:0/107355235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8481075a0 msgr2=0x7fe84819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8481075a0 0x7fe84819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 -- 172.21.15.77:0/107355235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe848102340 msgr2=0x7fe8481979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe848102340 0x7fe8481979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 -- 172.21.15.77:0/107355235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84819f5f0 con 0x7fe848104c70 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe847fff640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe848102340 0x7fe8481979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8477fe640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848104c70 0x7fe848197f10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00e9c0 tx=0x7fe83c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8457fa640 1 -- 172.21.15.77:0/107355235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c00cd80 con 0x7fe848104c70 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84819f8e0 con 0x7fe848104c70 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8457fa640 1 -- 172.21.15.77:0/107355235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe83c0054d0 con 0x7fe848104c70 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe8457fa640 1 -- 172.21.15.77:0/107355235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c010430 con 0x7fe848104c70 2024-03-16T09:42:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.847+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84819fe80 con 0x7fe848104c70 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.849+0000 7fe8457fa640 1 -- 172.21.15.77:0/107355235 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe83c004fd0 con 0x7fe848104c70 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 Processor -- start 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 -- start start 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd01760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd06830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffd09330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5623ffc66a40 con 0x5623ffd06ea0 2024-03-16T09:42:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5623ffc0f920 con 0x5623ffd01340 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8515b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5623ffc62ae0 con 0x5623ffd01d30 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8457fa640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe8300a28a0 0x7fe8300a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe8477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffd09330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe84cf63640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd01760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe847fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd06830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe847fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd06830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:57336/0 (socket says 172.21.15.77:57336) 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.850+0000 7fe84cf63640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd01760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:36392/0 (socket says 172.21.15.77:36392) 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe847fff640 1 -- 172.21.15.77:0/4017852416 learned_addr learned my addr 172.21.15.77:0/4017852416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe84cf63640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe8300a28a0 0x7fe8300a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe8457fa640 1 -- 172.21.15.77:0/107355235 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe83c0cee80 con 0x7fe848104c70 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe847fff640 1 -- 172.21.15.77:0/4017852416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 msgr2=0x5623ffd01760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe847fff640 1 --2- 172.21.15.77:0/4017852416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd01760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe847fff640 1 -- 172.21.15.77:0/4017852416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 msgr2=0x5623ffd09330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe847fff640 1 --2- 172.21.15.77:0/4017852416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffd09330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe847fff640 1 -- 172.21.15.77:0/4017852416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623ffc27460 con 0x5623ffd01d30 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe84cf63640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe8300a28a0 0x7fe8300a4d60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5623ffd02590 tx=0x7fe840009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe847fff640 1 --2- 172.21.15.77:0/4017852416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd06830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe838009900 tx=0x7fe8380095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9abf27b10c937670 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe82e7fc640 1 -- 172.21.15.77:0/4017852416 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838045020 con 0x5623ffd01d30 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe82e7fc640 1 -- 172.21.15.77:0/4017852416 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe83802cc90 con 0x5623ffd01d30 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.851+0000 7fe82e7fc640 1 -- 172.21.15.77:0/4017852416 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8380048c0 con 0x5623ffd01d30 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 -- 172.21.15.77:0/4017852416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 msgr2=0x5623ffd06830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 --2- 172.21.15.77:0/4017852416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd06830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe838009900 tx=0x7fe8380095f0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 -- 172.21.15.77:0/4017852416 shutdown_connections 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 --2- 172.21.15.77:0/4017852416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffd09330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 --2- 172.21.15.77:0/4017852416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd06830 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 --2- 172.21.15.77:0/4017852416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd01760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 -- 172.21.15.77:0/4017852416 >> 172.21.15.77:0/4017852416 conn(0x5623ffcfc8b0 msgr2=0x5623ffcfccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 -- 172.21.15.77:0/4017852416 shutdown_connections 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 -- 172.21.15.77:0/4017852416 wait complete. 2024-03-16T09:42:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.852+0000 7fe8515b5740 1 -- start start 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 1 Processor -- start 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd0b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01d30 0x5623ffd0bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd12b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5623ffd09df0 con 0x5623ffd01d30 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5623ffc22d70 con 0x5623ffd01340 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5623ffd09f70 con 0x5623ffd06ea0 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8515b5740 10 client.0 _ll_register_callbacks cb 0x5623ff97aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01d30 0x5623ffd0bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01d30 0x5623ffd0bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58576/0 (socket says 172.21.15.77:58576) 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 -- 172.21.15.77:0/1600515622 learned_addr learned my addr 172.21.15.77:0/1600515622 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe847fff640 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd0b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe84cf63640 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd12b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 -- 172.21.15.77:0/1600515622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 msgr2=0x5623ffd12b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd12b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 -- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 msgr2=0x5623ffd0b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd0b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.853+0000 7fe8477fe640 1 -- 172.21.15.77:0/1600515622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623ff97b210 con 0x5623ffd01d30 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8477fe640 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01d30 0x5623ffd0bac0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00c6a0 tx=0x7fe83c06b2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8feaabfcb2346b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8277fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8277fe640 1 -- 172.21.15.77:0/1600515622 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c09bed0 con 0x5623ffd01d30 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8277fe640 1 -- 172.21.15.77:0/1600515622 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe83c099ad0 con 0x5623ffd01d30 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe83802cc90 con 0x5623ffd01d30 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8277fe640 1 -- 172.21.15.77:0/1600515622 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe83c09b7d0 con 0x5623ffd01d30 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 10 client.15435 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5623ffa47620 con 0x5623ffd01d30 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8277fe640 1 -- 172.21.15.77:0/1600515622 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe83802cc90 con 0x5623ffd01d30 2024-03-16T09:42:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8277fe640 1 -- 172.21.15.77:0/1600515622 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe83c0baa50 con 0x5623ffd01d30 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 10 client.15435 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 10 client.15435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 20 client.15435 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd173e0 0x5623ffd19810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 4 client.15435 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623ffc73f80 con 0x5623ffd173e0 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.854+0000 7fe847fff640 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd173e0 0x5623ffd19810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.855+0000 7fe847fff640 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd173e0 0x5623ffd19810 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b43d425572c4a06 server_cookie=b23f9fbd1524a7a9 in_seq=0 out_seq=0 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.855+0000 7fe8277fe640 10 client.15435 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.855+0000 7fe8277fe640 1 -- 172.21.15.77:0/1600515622 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5623ffc73f80 con 0x5623ffd173e0 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.855+0000 7fe8277fe640 10 client.15435 handle_command_reply: tid=0 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.855+0000 7fe8515b5740 1 client.15435 shutdown 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01d30 msgr2=0x5623ffd0bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01d30 0x5623ffd0bac0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00c6a0 tx=0x7fe83c06b2f0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 shutdown_connections 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd173e0 0x5623ffd19810 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd12b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01d30 0x5623ffd0bac0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8277fe640 0 client.15435 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 --2- 172.21.15.77:0/1600515622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffd0b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 >> 172.21.15.77:0/1600515622 conn(0x5623ffcfc8b0 msgr2=0x5623ffcfbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 shutdown_connections 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 1 -- 172.21.15.77:0/1600515622 wait complete. 2024-03-16T09:42:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.856+0000 7fe8515b5740 20 client.15435 trim_cache size 0 max 16384 2024-03-16T09:42:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 Processor -- start 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 -- start start 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffcfb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd12ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffcfb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe8481ab400 con 0x5623ffd06ea0 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe8481a8cd0 con 0x5623ffd01340 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8515b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8481a6b20 con 0x5623ffd01d30 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffcfb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffcfb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58590/0 (socket says 172.21.15.77:58590) 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 -- 172.21.15.77:0/3711460271 learned_addr learned my addr 172.21.15.77:0/3711460271 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe847fff640 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffcfb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe8477fe640 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd12ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 -- 172.21.15.77:0/3711460271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 msgr2=0x5623ffd12ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd12ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 -- 172.21.15.77:0/3711460271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 msgr2=0x5623ffcfb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffcfb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.871+0000 7fe84cf63640 1 -- 172.21.15.77:0/3711460271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623ffcee140 con 0x5623ffd06ea0 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe84cf63640 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffcfb860 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe840009f10 tx=0x7fe840005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e058950e3029cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe82dffb640 1 -- 172.21.15.77:0/3711460271 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840011d70 con 0x5623ffd06ea0 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe82dffb640 1 -- 172.21.15.77:0/3711460271 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe84000fcc0 con 0x5623ffd06ea0 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe82dffb640 1 -- 172.21.15.77:0/3711460271 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840020c50 con 0x5623ffd06ea0 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe8515b5740 1 -- 172.21.15.77:0/3711460271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 msgr2=0x5623ffcfb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe8515b5740 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffcfb860 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe840009f10 tx=0x7fe840005bd0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe8515b5740 1 -- 172.21.15.77:0/3711460271 shutdown_connections 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe8515b5740 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd06ea0 0x5623ffcfb860 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe8515b5740 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd01d30 0x5623ffd12ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe8515b5740 1 --2- 172.21.15.77:0/3711460271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01340 0x5623ffcfb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.872+0000 7fe8515b5740 1 -- 172.21.15.77:0/3711460271 >> 172.21.15.77:0/3711460271 conn(0x5623ffcfc8b0 msgr2=0x5623ffcf7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 -- 172.21.15.77:0/3711460271 shutdown_connections 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 -- 172.21.15.77:0/3711460271 wait complete. 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 -- start start 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 Processor -- start 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01340 0x5623ffd12490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01d30 0x5623ffd129d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd10aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe8481a7190 con 0x5623ffd01340 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe8481a8600 con 0x5623ffd01d30 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8481a7860 con 0x5623ffd06ea0 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8515b5740 10 client.0 _ll_register_callbacks cb 0x7fe83c00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe847fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01340 0x5623ffd12490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe847fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01340 0x5623ffd12490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58598/0 (socket says 172.21.15.77:58598) 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe847fff640 1 -- 172.21.15.77:0/460897160 learned_addr learned my addr 172.21.15.77:0/460897160 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe8477fe640 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01d30 0x5623ffd129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.873+0000 7fe84cf63640 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd10aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe847fff640 1 -- 172.21.15.77:0/460897160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 msgr2=0x5623ffd10aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe847fff640 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd10aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe847fff640 1 -- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01d30 msgr2=0x5623ffd129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe847fff640 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01d30 0x5623ffd129d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe847fff640 1 -- 172.21.15.77:0/460897160 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623ffd112d0 con 0x5623ffd01340 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe847fff640 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01340 0x5623ffd12490 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe838005ab0 tx=0x7fe83802c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f37190a97d16b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8277fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8277fe640 1 -- 172.21.15.77:0/460897160 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83805a060 con 0x5623ffd01340 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8277fe640 1 -- 172.21.15.77:0/460897160 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe8380041c0 con 0x5623ffd01340 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe84000fcc0 con 0x5623ffd01340 2024-03-16T09:42:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8277fe640 1 -- 172.21.15.77:0/460897160 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe838045040 con 0x5623ffd01340 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8515b5740 10 client.15441 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5623ffd7e350 con 0x5623ffd01340 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8277fe640 1 -- 172.21.15.77:0/460897160 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe84000fcc0 con 0x5623ffd01340 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.874+0000 7fe8277fe640 1 -- 172.21.15.77:0/460897160 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe838041480 con 0x5623ffd01340 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8515b5740 10 client.15441 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8515b5740 10 client.15441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8515b5740 20 client.15441 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8515b5740 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd7e850 0x5623ffd80c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8515b5740 4 client.15441 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5623ffc6e2d0 con 0x5623ffd7e850 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8477fe640 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd7e850 0x5623ffd80c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8277fe640 10 client.15441 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.875+0000 7fe8477fe640 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd7e850 0x5623ffd80c40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8e64721e8fa5464 server_cookie=f0ea960dae9c11a8 in_seq=0 out_seq=0 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8277fe640 1 -- 172.21.15.77:0/460897160 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5623ffc6e2d0 con 0x5623ffd7e850 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8277fe640 10 client.15441 handle_command_reply: tid=0 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 client.15441 shutdown 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01340 msgr2=0x5623ffd12490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01340 0x5623ffd12490 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe838005ab0 tx=0x7fe83802c3e0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 shutdown_connections 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5623ffd7e850 0x5623ffd80c40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5623ffd06ea0 0x5623ffd10aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5623ffd01d30 0x5623ffd129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8277fe640 0 client.15441 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 --2- 172.21.15.77:0/460897160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623ffd01340 0x5623ffd12490 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 >> 172.21.15.77:0/460897160 conn(0x5623ffcfc8b0 msgr2=0x5623ffcfb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 shutdown_connections 2024-03-16T09:42:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 1 -- 172.21.15.77:0/460897160 wait complete. 2024-03-16T09:42:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.881+0000 7fe8515b5740 20 client.15441 trim_cache size 0 max 16384 2024-03-16T09:42:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:42:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:42:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:42:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe8300a28a0 msgr2=0x7fe8300a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe8300a28a0 0x7fe8300a4d60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5623ffd02590 tx=0x7fe840009500 comp rx=0 tx=0).stop 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848104c70 msgr2=0x7fe848197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848104c70 0x7fe848197f10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00e9c0 tx=0x7fe83c00ee80 comp rx=0 tx=0).stop 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 shutdown_connections 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fe8300a28a0 0x7fe8300a4d60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8481075a0 0x7fe84819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe848104c70 0x7fe848197f10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 --2- 172.21.15.77:0/107355235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe848102340 0x7fe8481979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 >> 172.21.15.77:0/107355235 conn(0x7fe8480fbcb0 msgr2=0x7fe848102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 shutdown_connections 2024-03-16T09:42:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:21.882+0000 7fe84e9c6640 1 -- 172.21.15.77:0/107355235 wait complete. 2024-03-16T09:42:23.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:22 smithi119 ceph-mon[35632]: pgmap v360: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-16T09:42:23.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:22 smithi077 ceph-mon[32171]: pgmap v360: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-16T09:42:23.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:22 smithi129 ceph-mon[35841]: pgmap v360: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-16T09:42:25.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:24 smithi119 ceph-mon[35632]: pgmap v361: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:25.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:24 smithi077 ceph-mon[32171]: pgmap v361: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:25.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:24 smithi129 ceph-mon[35841]: pgmap v361: 97 pgs: 97 active+clean; 659 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:25.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:42:25.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.039+0000 7f9eff7b9640 1 -- 172.21.15.77:0/3743956289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8102b50 msgr2=0x7f9ef8102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.039+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/3743956289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8102b50 0x7f9ef8102fd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009930 tx=0x7f9ee802bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.039+0000 7f9eff7b9640 1 -- 172.21.15.77:0/3743956289 shutdown_connections 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/3743956289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef8103510 0x7f9ef8109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/3743956289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8102b50 0x7f9ef8102fd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/3743956289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef80ffae0 0x7f9ef80fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 -- 172.21.15.77:0/3743956289 >> 172.21.15.77:0/3743956289 conn(0x7f9ef80fb410 msgr2=0x7f9ef80fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 -- 172.21.15.77:0/3743956289 shutdown_connections 2024-03-16T09:42:26.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 -- 172.21.15.77:0/3743956289 wait complete. 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 Processor -- start 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 -- start start 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef80ffae0 0x7f9ef8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef8102b50 0x7f9ef8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8103510 0x7f9ef819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d320 con 0x7f9ef8103510 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d1a0 con 0x7f9ef8102b50 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9eff7b9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef810d4a0 con 0x7f9ef80ffae0 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9efdd56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8103510 0x7f9ef819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9efdd56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8103510 0x7f9ef819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36694/0 (socket says 172.21.15.77:36694) 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9efdd56640 1 -- 172.21.15.77:0/1122608448 learned_addr learned my addr 172.21.15.77:0/1122608448 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9efcd54640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef8102b50 0x7f9ef8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.040+0000 7f9efd555640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef80ffae0 0x7f9ef8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9efdd56640 1 -- 172.21.15.77:0/1122608448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef80ffae0 msgr2=0x7f9ef8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9efdd56640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef80ffae0 0x7f9ef8197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9efdd56640 1 -- 172.21.15.77:0/1122608448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef8102b50 msgr2=0x7f9ef8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9efdd56640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef8102b50 0x7f9ef8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9efdd56640 1 -- 172.21.15.77:0/1122608448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef819f660 con 0x7f9ef8103510 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9efdd56640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8103510 0x7f9ef819eff0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000eb10 tx=0x7f9ef000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9eee7fc640 1 -- 172.21.15.77:0/1122608448 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0004b20 con 0x7f9ef8103510 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9eee7fc640 1 -- 172.21.15.77:0/1122608448 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ef0004cc0 con 0x7f9ef8103510 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef819f950 con 0x7f9ef8103510 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9eee7fc640 1 -- 172.21.15.77:0/1122608448 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0010690 con 0x7f9ef8103510 2024-03-16T09:42:26.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.041+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef819fe70 con 0x7f9ef8103510 2024-03-16T09:42:26.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.043+0000 7f9eee7fc640 1 -- 172.21.15.77:0/1122608448 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9ef0005020 con 0x7f9ef8103510 2024-03-16T09:42:26.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 Processor -- start 2024-03-16T09:42:26.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9eee7fc640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9ee40a2880 0x7f9ee40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 -- start start 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10a0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10a5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f10a8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9efcd54640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9ee40a2880 0x7f9ee40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9eee7fc640 1 -- 172.21.15.77:0/1122608448 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9ef00cef60 con 0x7f9ef8103510 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d1f1005500 con 0x55d1f10a5cd0 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fadbe0 con 0x55d1f10a0b60 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9f023a3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d1f10015a0 con 0x55d1f10a0170 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9efdd56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f10a8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9efd555640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9efdd56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f10a8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36698/0 (socket says 172.21.15.77:36698) 2024-03-16T09:42:26.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9efdd56640 1 -- 172.21.15.77:0/4261214788 learned_addr learned my addr 172.21.15.77:0/4261214788 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.044+0000 7f9efcd54640 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efdd56640 1 -- 172.21.15.77:0/4261214788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 msgr2=0x55d1f10a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efdd56640 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efdd56640 1 -- 172.21.15.77:0/4261214788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 msgr2=0x55d1f10a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efdd56640 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efdd56640 1 -- 172.21.15.77:0/4261214788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f0fc14b0 con 0x55d1f10a5cd0 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efd555640 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efcd54640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9ee40a2880 0x7f9ee40a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009900 tx=0x7f9ee802cb70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9efdd56640 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f10a8160 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00a3b80 tx=0x7f9ef009b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca87b1e9fb4e36c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9ee37fe640 1 -- 172.21.15.77:0/4261214788 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00b4070 con 0x55d1f10a5cd0 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9ee37fe640 1 -- 172.21.15.77:0/4261214788 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ef009b910 con 0x55d1f10a5cd0 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.045+0000 7f9ee37fe640 1 -- 172.21.15.77:0/4261214788 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00a3e40 con 0x55d1f10a5cd0 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- 172.21.15.77:0/4261214788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 msgr2=0x55d1f10a8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f10a8160 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00a3b80 tx=0x7f9ef009b450 comp rx=0 tx=0).stop 2024-03-16T09:42:26.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- 172.21.15.77:0/4261214788 shutdown_connections 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f10a8160 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 --2- 172.21.15.77:0/4261214788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- 172.21.15.77:0/4261214788 >> 172.21.15.77:0/4261214788 conn(0x55d1f109b6e0 msgr2=0x55d1f109bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- 172.21.15.77:0/4261214788 shutdown_connections 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- 172.21.15.77:0/4261214788 wait complete. 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- start start 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 Processor -- start 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10aac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d1f10a8c80 con 0x55d1f10a0b60 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0fc1160 con 0x55d1f10a5cd0 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d1f10a8e00 con 0x55d1f10a0170 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9f023a3740 10 client.0 _ll_register_callbacks cb 0x55d1f0fa5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.046+0000 7f9efcd54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10aac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36714/0 (socket says 172.21.15.77:36714) 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 -- 172.21.15.77:0/1110939816 learned_addr learned my addr 172.21.15.77:0/1110939816 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efd555640 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efdd56640 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 -- 172.21.15.77:0/1110939816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 msgr2=0x55d1f10aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 -- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 msgr2=0x55d1f10b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 -- 172.21.15.77:0/1110939816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f0de37d0 con 0x55d1f10a0b60 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9efcd54640 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10aac80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8035000 tx=0x7f9ee802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4bd0608d8c3fc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9ee0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/1110939816 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8050070 con 0x55d1f10a0b60 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/1110939816 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9ee80048d0 con 0x55d1f10a0b60 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ef009b910 con 0x55d1f10a0b60 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/1110939816 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9ee805c050 con 0x55d1f10a0b60 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9f023a3740 10 client.15453 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d1f0d193f0 con 0x55d1f10a0b60 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.047+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/1110939816 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ef009b910 con 0x55d1f10a0b60 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/1110939816 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9ee8040040 con 0x55d1f10a0b60 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9f023a3740 10 client.15453 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9f023a3740 10 client.15453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9f023a3740 20 client.15453 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9f023a3740 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f10b6220 0x55d1f10b8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9f023a3740 4 client.15453 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1f1012a40 con 0x55d1f10b6220 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9efd555640 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f10b6220 0x55d1f10b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9ee0ff9640 10 client.15453 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.048+0000 7f9efd555640 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f10b6220 0x55d1f10b8680 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef96a4d757eb3ea server_cookie=8e64fd8dd6f352a7 in_seq=0 out_seq=0 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/1110939816 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d1f1012a40 con 0x55d1f10b6220 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9ee0ff9640 10 client.15453 handle_command_reply: tid=0 2024-03-16T09:42:26.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 client.15453 shutdown 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 msgr2=0x55d1f10aac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10aac80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8035000 tx=0x7f9ee802bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 shutdown_connections 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f10b6220 0x55d1f10b8680 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10aac80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 --2- 172.21.15.77:0/1110939816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 >> 172.21.15.77:0/1110939816 conn(0x55d1f109b6e0 msgr2=0x55d1f109a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9ee0ff9640 0 client.15453 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 shutdown_connections 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 1 -- 172.21.15.77:0/1110939816 wait complete. 2024-03-16T09:42:26.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.049+0000 7f9f023a3740 20 client.15453 trim_cache size 0 max 16384 2024-03-16T09:42:26.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.066+0000 7f9f023a3740 1 Processor -- start 2024-03-16T09:42:26.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.066+0000 7f9f023a3740 1 -- start start 2024-03-16T09:42:26.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f109a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f109a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9f023a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81ab490 con 0x55d1f10a5cd0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9f023a3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81a8d60 con 0x55d1f10a0b60 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9f023a3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81a6bb0 con 0x55d1f10a0170 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f109a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efd555640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f109a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efcd54640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f109a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36718/0 (socket says 172.21.15.77:36718) 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 -- 172.21.15.77:0/2880108362 learned_addr learned my addr 172.21.15.77:0/2880108362 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 -- 172.21.15.77:0/2880108362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 msgr2=0x55d1f109a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f109a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 -- 172.21.15.77:0/2880108362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 msgr2=0x55d1f10b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 -- 172.21.15.77:0/2880108362 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f0cd90f0 con 0x55d1f10a5cd0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efcd54640 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9efdd56640 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f109a5b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009270 tx=0x7f9ef0009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ef8ccbba4c9169 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9ee2ffd640 1 -- 172.21.15.77:0/2880108362 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00b4040 con 0x55d1f10a5cd0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9ee2ffd640 1 -- 172.21.15.77:0/2880108362 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ef0099c10 con 0x55d1f10a5cd0 2024-03-16T09:42:26.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.067+0000 7f9ee2ffd640 1 -- 172.21.15.77:0/2880108362 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0005570 con 0x55d1f10a5cd0 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 -- 172.21.15.77:0/2880108362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 msgr2=0x55d1f109a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f109a5b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009270 tx=0x7f9ef0009200 comp rx=0 tx=0).stop 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 -- 172.21.15.77:0/2880108362 shutdown_connections 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a5cd0 0x55d1f109a5b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a0b60 0x55d1f10b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2880108362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f109a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 -- 172.21.15.77:0/2880108362 >> 172.21.15.77:0/2880108362 conn(0x55d1f109b6e0 msgr2=0x55d1f10aa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 -- 172.21.15.77:0/2880108362 shutdown_connections 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 -- 172.21.15.77:0/2880108362 wait complete. 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 -- start start 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.068+0000 7f9f023a3740 1 Processor -- start 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10afde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81a7220 con 0x55d1f10a0b60 2024-03-16T09:42:26.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81a8690 con 0x55d1f10a5cd0 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef81a78f0 con 0x55d1f10a0170 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 10 client.0 _ll_register_callbacks cb 0x7f9ee8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36734/0 (socket says 172.21.15.77:36734) 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 -- 172.21.15.77:0/2209613581 learned_addr learned my addr 172.21.15.77:0/2209613581 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efd555640 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efdd56640 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 -- 172.21.15.77:0/2209613581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 msgr2=0x55d1f10af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10af8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 -- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 msgr2=0x55d1f10b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 -- 172.21.15.77:0/2209613581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f0fcc9d0 con 0x55d1f10a0b60 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9efcd54640 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10afde0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009930 tx=0x7f9ee8034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41495a0b684a7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9ee0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:26.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/2209613581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8050040 con 0x55d1f10a0b60 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/2209613581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9ee8003800 con 0x55d1f10a0b60 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ef0099c10 con 0x55d1f10a0b60 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/2209613581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9ee8049070 con 0x55d1f10a0b60 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 10 client.15459 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.069+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d1f111c840 con 0x55d1f10a0b60 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/2209613581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ef0099c10 con 0x55d1f10a0b60 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/2209613581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9ee80346e0 con 0x55d1f10a0b60 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9f023a3740 10 client.15459 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9f023a3740 10 client.15459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9f023a3740 20 client.15459 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f111d1e0 0x55d1f111f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9f023a3740 4 client.15459 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d1f100cd90 con 0x55d1f111d1e0 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9efd555640 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f111d1e0 0x55d1f111f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9efd555640 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f111d1e0 0x55d1f111f600 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51a78db757f537e6 server_cookie=4c93452c7ef91982 in_seq=0 out_seq=0 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9ee0ff9640 10 client.15459 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9ee0ff9640 1 -- 172.21.15.77:0/2209613581 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d1f100cd90 con 0x55d1f111d1e0 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9ee0ff9640 10 client.15459 handle_command_reply: tid=0 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.070+0000 7f9f023a3740 1 client.15459 shutdown 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 msgr2=0x55d1f10afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10afde0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009930 tx=0x7f9ee8034670 comp rx=0 tx=0).stop 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 shutdown_connections 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55d1f111d1e0 0x55d1f111f600 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1f10a5cd0 0x55d1f10b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9ee0ff9640 0 client.15459 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1f10a0b60 0x55d1f10afde0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 --2- 172.21.15.77:0/2209613581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d1f10a0170 0x55d1f10af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 >> 172.21.15.77:0/2209613581 conn(0x55d1f109b6e0 msgr2=0x55d1f109e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 shutdown_connections 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 1 -- 172.21.15.77:0/2209613581 wait complete. 2024-03-16T09:42:26.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9f023a3740 20 client.15459 trim_cache size 0 max 16384 2024-03-16T09:42:26.081 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:42:26.081 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T09:42:26.081 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T09:42:26.081 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:42:26.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9ee40a2880 msgr2=0x7f9ee40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9ee40a2880 0x7f9ee40a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009900 tx=0x7f9ee802cb70 comp rx=0 tx=0).stop 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8103510 msgr2=0x7f9ef819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8103510 0x7f9ef819eff0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000eb10 tx=0x7f9ef000c2d0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 shutdown_connections 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9ee40a2880 0x7f9ee40a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef8103510 0x7f9ef819eff0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef8102b50 0x7f9ef8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 --2- 172.21.15.77:0/1122608448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef80ffae0 0x7f9ef8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 >> 172.21.15.77:0/1122608448 conn(0x7f9ef80fb410 msgr2=0x7f9ef8105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 shutdown_connections 2024-03-16T09:42:26.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.071+0000 7f9eff7b9640 1 -- 172.21.15.77:0/1122608448 wait complete. 2024-03-16T09:42:26.085 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T09:42:26.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:42:26.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 -- 172.21.15.77:0/1655090184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd801018f0 msgr2=0x7fdd80101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/1655090184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd801018f0 0x7fdd80101d70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009860 tx=0x7fdd7002bb90 comp rx=0 tx=0).stop 2024-03-16T09:42:26.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 -- 172.21.15.77:0/1655090184 shutdown_connections 2024-03-16T09:42:26.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/1655090184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd801022b0 0x7fdd80108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/1655090184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd801018f0 0x7fdd80101d70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/1655090184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd80100700 0x7fdd80100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 -- 172.21.15.77:0/1655090184 >> 172.21.15.77:0/1655090184 conn(0x7fdd800fbc70 msgr2=0x7fdd800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 -- 172.21.15.77:0/1655090184 shutdown_connections 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.183+0000 7fdd84f5b640 1 -- 172.21.15.77:0/1655090184 wait complete. 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 Processor -- start 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 -- start start 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd80100700 0x7fdd8006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd801018f0 0x7fdd8019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd801022b0 0x7fdd8019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8010c210 con 0x7fdd801022b0 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8010c090 con 0x7fdd80100700 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd84f5b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8010c390 con 0x7fdd801018f0 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd801022b0 0x7fdd8019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd801022b0 0x7fdd8019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36744/0 (socket says 172.21.15.77:36744) 2024-03-16T09:42:26.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 -- 172.21.15.77:0/3074119161 learned_addr learned my addr 172.21.15.77:0/3074119161 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7e59c640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd80100700 0x7fdd8006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7dd9b640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd801018f0 0x7fdd8019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 -- 172.21.15.77:0/3074119161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd801018f0 msgr2=0x7fdd8019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd801018f0 0x7fdd8019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 -- 172.21.15.77:0/3074119161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd80100700 msgr2=0x7fdd8006a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd80100700 0x7fdd8006a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7ed9d640 1 -- 172.21.15.77:0/3074119161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8019f690 con 0x7fdd801022b0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.184+0000 7fdd7e59c640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd80100700 0x7fdd8006a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.185+0000 7fdd7ed9d640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd801022b0 0x7fdd8019ed80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800eb10 tx=0x7fdd7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.185+0000 7fdd777fe640 1 -- 172.21.15.77:0/3074119161 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78004b20 con 0x7fdd801022b0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.185+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8019f920 con 0x7fdd801022b0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.185+0000 7fdd777fe640 1 -- 172.21.15.77:0/3074119161 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd78004cc0 con 0x7fdd801022b0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.185+0000 7fdd777fe640 1 -- 172.21.15.77:0/3074119161 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78010690 con 0x7fdd801022b0 2024-03-16T09:42:26.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.185+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8019fdf0 con 0x7fdd801022b0 2024-03-16T09:42:26.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.186+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd80107a90 con 0x7fdd801022b0 2024-03-16T09:42:26.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.187+0000 7fdd777fe640 1 -- 172.21.15.77:0/3074119161 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdd78005000 con 0x7fdd801022b0 2024-03-16T09:42:26.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.187+0000 7fdd777fe640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdd6c0a2800 0x7fdd6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.188+0000 7fdd7e59c640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdd6c0a2800 0x7fdd6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.188+0000 7fdd777fe640 1 -- 172.21.15.77:0/3074119161 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdd780cf1e0 con 0x7fdd801022b0 2024-03-16T09:42:26.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.188+0000 7fdd7e59c640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdd6c0a2800 0x7fdd6c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdd680096c0 tx=0x7fdd68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.191+0000 7fdd777fe640 1 -- 172.21.15.77:0/3074119161 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd78053740 con 0x7fdd801022b0 2024-03-16T09:42:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.406+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd8002d370 con 0x7fdd801022b0 2024-03-16T09:42:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.407+0000 7fdd777fe640 1 -- 172.21.15.77:0/3074119161 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7fdd7809c290 con 0x7fdd801022b0 2024-03-16T09:42:26.410 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:42:26.410 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:42:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:42:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdd6c0a2800 msgr2=0x7fdd6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdd6c0a2800 0x7fdd6c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdd680096c0 tx=0x7fdd68009260 comp rx=0 tx=0).stop 2024-03-16T09:42:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd801022b0 msgr2=0x7fdd8019ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd801022b0 0x7fdd8019ed80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800eb10 tx=0x7fdd7800c2d0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 shutdown_connections 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fdd6c0a2800 0x7fdd6c0a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd801022b0 0x7fdd8019ed80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd801018f0 0x7fdd8019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 --2- 172.21.15.77:0/3074119161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd80100700 0x7fdd8006a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 >> 172.21.15.77:0/3074119161 conn(0x7fdd800fbc70 msgr2=0x7fdd80104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 shutdown_connections 2024-03-16T09:42:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.409+0000 7fdd84f5b640 1 -- 172.21.15.77:0/3074119161 wait complete. 2024-03-16T09:42:26.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- 172.21.15.77:0/3023876886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181018d0 msgr2=0x7fbe18101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 --2- 172.21.15.77:0/3023876886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181018d0 0x7fbe18101d50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009900 tx=0x7fbe0802bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- 172.21.15.77:0/3023876886 shutdown_connections 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 --2- 172.21.15.77:0/3023876886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe18102290 0x7fbe18108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 --2- 172.21.15.77:0/3023876886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181018d0 0x7fbe18101d50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 --2- 172.21.15.77:0/3023876886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe181006e0 0x7fbe18100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- 172.21.15.77:0/3023876886 >> 172.21.15.77:0/3023876886 conn(0x7fbe180fbc70 msgr2=0x7fbe180fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- 172.21.15.77:0/3023876886 shutdown_connections 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- 172.21.15.77:0/3023876886 wait complete. 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 Processor -- start 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- start start 2024-03-16T09:42:26.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181006e0 0x7fbe180694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe181018d0 0x7fbe1819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe18102290 0x7fbe1819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810c210 con 0x7fbe181006e0 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810c090 con 0x7fbe18102290 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1f705640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810c390 con 0x7fbe181018d0 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1d4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181006e0 0x7fbe180694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1d4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181006e0 0x7fbe180694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36752/0 (socket says 172.21.15.77:36752) 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.526+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/2985139566 learned_addr learned my addr 172.21.15.77:0/2985139566 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe181018d0 0x7fbe1819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe18102290 0x7fbe1819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/2985139566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe181018d0 msgr2=0x7fbe1819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe181018d0 0x7fbe1819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/2985139566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe18102290 msgr2=0x7fbe1819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe18102290 0x7fbe1819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/2985139566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1819f670 con 0x7fbe181006e0 2024-03-16T09:42:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181006e0 0x7fbe180694d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400e9c0 tx=0x7fbe1400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/2985139566 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1400cd80 con 0x7fbe181006e0 2024-03-16T09:42:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/2985139566 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe140035f0 con 0x7fbe181006e0 2024-03-16T09:42:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/2985139566 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14010690 con 0x7fbe181006e0 2024-03-16T09:42:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1819f960 con 0x7fbe181006e0 2024-03-16T09:42:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.527+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe1819fe30 con 0x7fbe181006e0 2024-03-16T09:42:26.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.529+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/2985139566 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbe140037a0 con 0x7fbe181006e0 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 Processor -- start 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 -- start start 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b162241c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b162246d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b1622473b0 0x55b162249840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b1621a6490 con 0x55b162241850 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b16214ec90 con 0x55b1622473b0 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe223c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b1621a2530 con 0x55b162242240 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe0e7fc640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbe040a2800 0x7fbe040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b162241c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b162241c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36764/0 (socket says 172.21.15.77:36764) 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 -- 172.21.15.77:0/1530628447 learned_addr learned my addr 172.21.15.77:0/1530628447 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b1622473b0 0x55b162249840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbe040a2800 0x7fbe040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe0e7fc640 1 -- 172.21.15.77:0/2985139566 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbe140cedd0 con 0x7fbe181006e0 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b162246d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 -- 172.21.15.77:0/1530628447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 msgr2=0x55b162246d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b162246d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 -- 172.21.15.77:0/1530628447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b1622473b0 msgr2=0x55b162249840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b1622473b0 0x55b162249840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1dca2640 1 -- 172.21.15.77:0/1530628447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b162209500 con 0x55b162241850 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.530+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b1622473b0 0x55b162249840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.531+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b162241c70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbe10011e20 tx=0x7fbe1000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a41b4df169255f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.531+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbe040a2800 0x7fbe040a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000fc90 tx=0x7fbe1001e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.531+0000 7fbe037fe640 1 -- 172.21.15.77:0/1530628447 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1001dcb0 con 0x55b162241850 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.531+0000 7fbe037fe640 1 -- 172.21.15.77:0/1530628447 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe1001de50 con 0x55b162241850 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.531+0000 7fbe037fe640 1 -- 172.21.15.77:0/1530628447 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe10004280 con 0x55b162241850 2024-03-16T09:42:26.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- 172.21.15.77:0/1530628447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 msgr2=0x55b162241c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b162241c70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbe10011e20 tx=0x7fbe1000f790 comp rx=0 tx=0).stop 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- 172.21.15.77:0/1530628447 shutdown_connections 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b1622473b0 0x55b162249840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b162246d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1530628447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b162241c70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- 172.21.15.77:0/1530628447 >> 172.21.15.77:0/1530628447 conn(0x55b16223cdc0 msgr2=0x55b16223d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- 172.21.15.77:0/1530628447 shutdown_connections 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- 172.21.15.77:0/1530628447 wait complete. 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- start start 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 Processor -- start 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b16224c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b162253260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b16224a2a0 con 0x55b162241850 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b162162230 con 0x55b162242240 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b16224a420 con 0x55b1622473b0 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.532+0000 7fbe223c5740 10 client.0 _ll_register_callbacks cb 0x55b16219e010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1d4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1cca0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b16224c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1dca2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b162253260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1d4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36780/0 (socket says 172.21.15.77:36780) 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/1121498658 learned_addr learned my addr 172.21.15.77:0/1121498658 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1cca0640 1 -- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 msgr2=0x55b162253260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b162253260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1cca0640 1 -- 172.21.15.77:0/1121498658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 msgr2=0x55b16224bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1cca0640 1 -- 172.21.15.77:0/1121498658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b162171100 con 0x55b162242240 2024-03-16T09:42:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b162253260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b16224c260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbe08000c00 tx=0x7fbe08004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9705212f47794caf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe00ff9640 1 -- 172.21.15.77:0/1121498658 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08005030 con 0x55b162242240 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe1001de50 con 0x55b162242240 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe00ff9640 1 -- 172.21.15.77:0/1121498658 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbe080051d0 con 0x55b162242240 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe00ff9640 1 -- 172.21.15.77:0/1121498658 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbe08051050 con 0x55b162242240 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe223c5740 10 client.25367 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.533+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b161eba3f0 con 0x55b162242240 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe00ff9640 1 -- 172.21.15.77:0/1121498658 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe1001de50 con 0x55b162242240 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe00ff9640 1 -- 172.21.15.77:0/1121498658 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbe080035f0 con 0x55b162242240 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe223c5740 10 client.25367 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe223c5740 10 client.25367 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe223c5740 20 client.25367 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b162257ab0 0x55b162259f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe223c5740 4 client.25367 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1621b39d0 con 0x55b162257ab0 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b162257ab0 0x55b162259f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe00ff9640 10 client.25367 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.534+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b162257ab0 0x55b162259f00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70c294d3aa68fb8b server_cookie=859bfaadf05fd4e4 in_seq=0 out_seq=0 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe00ff9640 1 -- 172.21.15.77:0/1121498658 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b1621b39d0 con 0x55b162257ab0 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe00ff9640 10 client.25367 handle_command_reply: tid=0 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 client.25367 shutdown 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 msgr2=0x55b16224c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b16224c260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbe08000c00 tx=0x7fbe08004270 comp rx=0 tx=0).stop 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 shutdown_connections 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b162253260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b162257ab0 0x55b162259f00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b16224c260 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 --2- 172.21.15.77:0/1121498658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe00ff9640 0 client.25367 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:26.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 >> 172.21.15.77:0/1121498658 conn(0x55b16223cdc0 msgr2=0x55b16223bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 shutdown_connections 2024-03-16T09:42:26.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 1 -- 172.21.15.77:0/1121498658 wait complete. 2024-03-16T09:42:26.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.535+0000 7fbe223c5740 20 client.25367 trim_cache size 0 max 16384 2024-03-16T09:42:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 Processor -- start 2024-03-16T09:42:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 -- start start 2024-03-16T09:42:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b16224c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1622473b0 0x55b162253220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181ab4e0 con 0x55b1622473b0 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181a8db0 con 0x55b162241850 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe223c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181a6c00 con 0x55b162242240 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1622473b0 0x55b162253220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1d4a1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1622473b0 0x55b162253220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36786/0 (socket says 172.21.15.77:36786) 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 -- 172.21.15.77:0/2236305229 learned_addr learned my addr 172.21.15.77:0/2236305229 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:39810/0 (socket says 172.21.15.77:39810) 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b16224c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 -- 172.21.15.77:0/2236305229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 msgr2=0x55b16224c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b16224c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 -- 172.21.15.77:0/2236305229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162241850 msgr2=0x55b16224bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 -- 172.21.15.77:0/2236305229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b16224b8c0 con 0x55b1622473b0 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.549+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1622473b0 0x55b162253220 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbe1001dbc0 tx=0x7fbe10011460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07f888a27bec7b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe02ffd640 1 -- 172.21.15.77:0/2236305229 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe10035020 con 0x55b1622473b0 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe02ffd640 1 -- 172.21.15.77:0/2236305229 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe1002b040 con 0x55b1622473b0 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe02ffd640 1 -- 172.21.15.77:0/2236305229 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1002a040 con 0x55b1622473b0 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 -- 172.21.15.77:0/2236305229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1622473b0 msgr2=0x55b162253220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1622473b0 0x55b162253220 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbe1001dbc0 tx=0x7fbe10011460 comp rx=0 tx=0).stop 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 -- 172.21.15.77:0/2236305229 shutdown_connections 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b1622473b0 0x55b162253220 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b162242240 0x55b16224c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 --2- 172.21.15.77:0/2236305229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162241850 0x55b16224bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 -- 172.21.15.77:0/2236305229 >> 172.21.15.77:0/2236305229 conn(0x55b16223cdc0 msgr2=0x55b16223d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 -- 172.21.15.77:0/2236305229 shutdown_connections 2024-03-16T09:42:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.550+0000 7fbe223c5740 1 -- 172.21.15.77:0/2236305229 wait complete. 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 -- start start 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 Processor -- start 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b1622504d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b1622bbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181a7270 con 0x55b162241850 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181a86e0 con 0x55b162242240 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181a7940 con 0x55b1622473b0 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe223c5740 10 client.0 _ll_register_callbacks cb 0x7fbe08009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36792/0 (socket says 172.21.15.77:36792) 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/3283350762 learned_addr learned my addr 172.21.15.77:0/3283350762 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b1622504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1dca2640 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b1622bbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 msgr2=0x55b1622bbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b1622bbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/3283350762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 msgr2=0x55b1622504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b1622504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 -- 172.21.15.77:0/3283350762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b161f847d0 con 0x55b162241850 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.551+0000 7fbe1d4a1640 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224ff90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbe1409b480 tx=0x7fbe14096100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fb350a12c22b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe00ff9640 1 -- 172.21.15.77:0/3283350762 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1409b780 con 0x55b162241850 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe00ff9640 1 -- 172.21.15.77:0/3283350762 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbe140860b0 con 0x55b162241850 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe1002b040 con 0x55b162241850 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe00ff9640 1 -- 172.21.15.77:0/3283350762 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbe1409f8d0 con 0x55b162241850 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 10 client.15483 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b1622bd1b0 con 0x55b162241850 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe00ff9640 1 -- 172.21.15.77:0/3283350762 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe1002b040 con 0x55b162241850 2024-03-16T09:42:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe00ff9640 1 -- 172.21.15.77:0/3283350762 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbe14097040 con 0x55b162241850 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 10 client.15483 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 10 client.15483 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 20 client.15483 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b1622bdae0 0x55b1622bff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 4 client.15483 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b1621add20 con 0x55b1622bdae0 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.552+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b1622bdae0 0x55b1622bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe00ff9640 10 client.15483 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe1cca0640 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b1622bdae0 0x55b1622bff00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36f6b27820a7518 server_cookie=777df7e48045585d in_seq=0 out_seq=0 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe00ff9640 1 -- 172.21.15.77:0/3283350762 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b1621add20 con 0x55b1622bdae0 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe00ff9640 10 client.15483 handle_command_reply: tid=0 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 client.15483 shutdown 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 msgr2=0x55b16224ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224ff90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbe1409b480 tx=0x7fbe14096100 comp rx=0 tx=0).stop 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 shutdown_connections 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1622473b0 0x55b1622bbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55b1622bdae0 0x55b1622bff00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b162242240 0x55b1622504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe00ff9640 0 client.15483 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 --2- 172.21.15.77:0/3283350762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b162241850 0x55b16224ff90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 >> 172.21.15.77:0/3283350762 conn(0x55b16223cdc0 msgr2=0x55b16224bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 shutdown_connections 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 1 -- 172.21.15.77:0/3283350762 wait complete. 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.553+0000 7fbe223c5740 20 client.15483 trim_cache size 0 max 16384 2024-03-16T09:42:26.563 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbe040a2800 msgr2=0x7fbe040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbe040a2800 0x7fbe040a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000fc90 tx=0x7fbe1001e000 comp rx=0 tx=0).stop 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181006e0 msgr2=0x7fbe180694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181006e0 0x7fbe180694d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400e9c0 tx=0x7fbe1400ee80 comp rx=0 tx=0).stop 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 shutdown_connections 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fbe040a2800 0x7fbe040a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe18102290 0x7fbe1819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe181018d0 0x7fbe1819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 --2- 172.21.15.77:0/2985139566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe181006e0 0x7fbe180694d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 >> 172.21.15.77:0/2985139566 conn(0x7fbe180fbc70 msgr2=0x7fbe18104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 shutdown_connections 2024-03-16T09:42:26.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.554+0000 7fbe1f705640 1 -- 172.21.15.77:0/2985139566 wait complete. 2024-03-16T09:42:26.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T09:42:26.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.668+0000 7f7a44fe6640 1 -- 172.21.15.77:0/1932211935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a40102290 msgr2=0x7f7a40108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.668+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/1932211935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a40102290 0x7f7a40108b20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009ca0 tx=0x7f7a3002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.668+0000 7f7a44fe6640 1 -- 172.21.15.77:0/1932211935 shutdown_connections 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.668+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/1932211935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a40102290 0x7f7a40108b20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.668+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/1932211935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a401018d0 0x7f7a40101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.668+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/1932211935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a401006e0 0x7f7a40100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.668+0000 7f7a44fe6640 1 -- 172.21.15.77:0/1932211935 >> 172.21.15.77:0/1932211935 conn(0x7f7a400fbc70 msgr2=0x7f7a400fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 -- 172.21.15.77:0/1932211935 shutdown_connections 2024-03-16T09:42:26.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 -- 172.21.15.77:0/1932211935 wait complete. 2024-03-16T09:42:26.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 Processor -- start 2024-03-16T09:42:26.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 -- start start 2024-03-16T09:42:26.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a401006e0 0x7f7a40197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a401018d0 0x7f7a40197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a40102290 0x7f7a4019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4010c210 con 0x7f7a401006e0 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4010c090 con 0x7f7a401018d0 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.669+0000 7f7a44fe6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4010c390 con 0x7f7a40102290 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a401006e0 0x7f7a40197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a401018d0 0x7f7a40197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a40102290 0x7f7a4019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a401018d0 0x7f7a40197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:39846/0 (socket says 172.21.15.77:39846) 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a401006e0 0x7f7a40197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36804/0 (socket says 172.21.15.77:36804) 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/3520235541 learned_addr learned my addr 172.21.15.77:0/3520235541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3ed9d640 1 -- 172.21.15.77:0/3520235541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a401018d0 msgr2=0x7f7a40197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3ed9d640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a401018d0 0x7f7a40197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3ed9d640 1 -- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a401006e0 msgr2=0x7f7a40197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3ed9d640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a401006e0 0x7f7a40197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3ed9d640 1 -- 172.21.15.77:0/3520235541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4019f630 con 0x7f7a40102290 2024-03-16T09:42:26.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a401006e0 0x7f7a40197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a3ed9d640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a40102290 0x7f7a4019efc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a3002c630 tx=0x7f7a300328a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a377fe640 1 -- 172.21.15.77:0/3520235541 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30041020 con 0x7f7a40102290 2024-03-16T09:42:26.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4019f8c0 con 0x7f7a40102290 2024-03-16T09:42:26.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a377fe640 1 -- 172.21.15.77:0/3520235541 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a30003e20 con 0x7f7a40102290 2024-03-16T09:42:26.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a377fe640 1 -- 172.21.15.77:0/3520235541 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30039040 con 0x7f7a40102290 2024-03-16T09:42:26.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.670+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a4019fdb0 con 0x7f7a40102290 2024-03-16T09:42:26.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.672+0000 7f7a377fe640 1 -- 172.21.15.77:0/3520235541 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7a30044020 con 0x7f7a40102290 2024-03-16T09:42:26.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 Processor -- start 2024-03-16T09:42:26.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 -- start start 2024-03-16T09:42:26.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238ca9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238caed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238cb1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c238c0e490 con 0x55c238caf3b0 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c238bb6c90 con 0x55c238caa240 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c238c0a530 con 0x55c238ca9850 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a377fe640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a2c0a2800 0x7f7a2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238cb1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238caed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238cb1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36816/0 (socket says 172.21.15.77:36816) 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/235659162 learned_addr learned my addr 172.21.15.77:0/235659162 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238caed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:39858/0 (socket says 172.21.15.77:39858) 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238ca9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3ed9d640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a2c0a2800 0x7f7a2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a377fe640 1 -- 172.21.15.77:0/3520235541 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7a300f0c40 con 0x7f7a40102290 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.673+0000 7f7a3e59c640 1 -- 172.21.15.77:0/235659162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 msgr2=0x55c238ca9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238ca9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a3e59c640 1 -- 172.21.15.77:0/235659162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 msgr2=0x55c238cb1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238cb1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a3e59c640 1 -- 172.21.15.77:0/235659162 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c238c71500 con 0x55c238caa240 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238cb1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a3ed9d640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a2c0a2800 0x7f7a2c0a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7a30006010 tx=0x7f7a30088350 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238caed40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009fa0 tx=0x7f7a380118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bc2111ef4dd86bd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a34ff9640 1 -- 172.21.15.77:0/235659162 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38004310 con 0x55c238caa240 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a34ff9640 1 -- 172.21.15.77:0/235659162 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a380044b0 con 0x55c238caa240 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.674+0000 7f7a34ff9640 1 -- 172.21.15.77:0/235659162 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a380035f0 con 0x55c238caa240 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- 172.21.15.77:0/235659162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 msgr2=0x55c238caed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238caed40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009fa0 tx=0x7f7a380118d0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- 172.21.15.77:0/235659162 shutdown_connections 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238cb1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238caed40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/235659162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238ca9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- 172.21.15.77:0/235659162 >> 172.21.15.77:0/235659162 conn(0x55c238ca4dc0 msgr2=0x55c238ca51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- 172.21.15.77:0/235659162 shutdown_connections 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- 172.21.15.77:0/235659162 wait complete. 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- start start 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 Processor -- start 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238ca9850 0x55c238cb3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238cb42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caf3b0 0x55c238cbb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c238cb2360 con 0x55c238ca9850 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c238bca230 con 0x55c238caa240 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c238cb24e0 con 0x55c238caf3b0 2024-03-16T09:42:26.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.675+0000 7f7a47bd0740 10 client.0 _ll_register_callbacks cb 0x55c238c06010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238ca9850 0x55c238cb3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238ca9850 0x55c238cb3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36828/0 (socket says 172.21.15.77:36828) 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/3216049182 learned_addr learned my addr 172.21.15.77:0/3216049182 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238cb42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caf3b0 0x55c238cbb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/3216049182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caf3b0 msgr2=0x55c238cbb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caf3b0 0x55c238cbb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 msgr2=0x55c238cb42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238cb42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/3216049182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c238bd9100 con 0x55c238ca9850 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238ca9850 0x55c238cb3da0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7a28011e40 tx=0x7f7a28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fce0ec6a32289fd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3216049182 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2800fe60 con 0x55c238ca9850 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3216049182 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7a28005020 con 0x55c238ca9850 2024-03-16T09:42:26.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a380044b0 con 0x55c238ca9850 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3216049182 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7a28005220 con 0x55c238ca9850 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a47bd0740 10 client.15495 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c2389223f0 con 0x55c238ca9850 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.676+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3216049182 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a380044b0 con 0x55c238ca9850 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3216049182 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7a280203f0 con 0x55c238ca9850 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a47bd0740 10 client.15495 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a47bd0740 10 client.15495 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a47bd0740 20 client.15495 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238cbfc30 0x55c238cc20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a47bd0740 4 client.15495 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c238c1b9d0 con 0x55c238cbfc30 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238cbfc30 0x55c238cc20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238cbfc30 0x55c238cc20d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a95222e7f9c170c server_cookie=94c41feb5d0920b7 in_seq=0 out_seq=0 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.677+0000 7f7a25ffb640 10 client.15495 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:26.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3216049182 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c238c1b9d0 con 0x55c238cbfc30 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a25ffb640 10 client.15495 handle_command_reply: tid=0 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 client.15495 shutdown 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238ca9850 msgr2=0x55c238cb3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238ca9850 0x55c238cb3da0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7a28011e40 tx=0x7f7a28004730 comp rx=0 tx=0).stop 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 shutdown_connections 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238cbfc30 0x55c238cc20d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caf3b0 0x55c238cbb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a25ffb640 0 client.15495 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caa240 0x55c238cb42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3216049182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238ca9850 0x55c238cb3da0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 >> 172.21.15.77:0/3216049182 conn(0x55c238ca4dc0 msgr2=0x55c238ca3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 shutdown_connections 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3216049182 wait complete. 2024-03-16T09:42:26.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.678+0000 7f7a47bd0740 20 client.15495 trim_cache size 0 max 16384 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.691+0000 7f7a47bd0740 1 Processor -- start 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.691+0000 7f7a47bd0740 1 -- start start 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238ca9850 0x55c238cb4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caa240 0x55c238cbb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238ca8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a401ab410 con 0x55c238caf3b0 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7a401a8ce0 con 0x55c238ca9850 2024-03-16T09:42:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7a401a6b30 con 0x55c238caa240 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238ca8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238ca8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36844/0 (socket says 172.21.15.77:36844) 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/2251689246 learned_addr learned my addr 172.21.15.77:0/2251689246 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caa240 0x55c238cbb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238ca9850 0x55c238cb4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/2251689246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caa240 msgr2=0x55c238cbb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caa240 0x55c238cbb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/2251689246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238ca9850 msgr2=0x55c238cb4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238ca9850 0x55c238cb4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 -- 172.21.15.77:0/2251689246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2389ec7d0 con 0x55c238caf3b0 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caa240 0x55c238cbb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238ca8730 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009de0 tx=0x7f7a28005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a22c6c899da8d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a27fff640 1 -- 172.21.15.77:0/2251689246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2802f030 con 0x55c238caf3b0 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a27fff640 1 -- 172.21.15.77:0/2251689246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a28030040 con 0x55c238caf3b0 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.692+0000 7f7a27fff640 1 -- 172.21.15.77:0/2251689246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28005620 con 0x55c238caf3b0 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- 172.21.15.77:0/2251689246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 msgr2=0x55c238ca8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238ca8730 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009de0 tx=0x7f7a28005970 comp rx=0 tx=0).stop 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- 172.21.15.77:0/2251689246 shutdown_connections 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caf3b0 0x55c238ca8730 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238caa240 0x55c238cbb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/2251689246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238ca9850 0x55c238cb4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- 172.21.15.77:0/2251689246 >> 172.21.15.77:0/2251689246 conn(0x55c238ca4dc0 msgr2=0x55c238ca8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- 172.21.15.77:0/2251689246 shutdown_connections 2024-03-16T09:42:26.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- 172.21.15.77:0/2251689246 wait complete. 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- start start 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 Processor -- start 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238cb8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caa240 0x55c238cb89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caf3b0 0x55c238cb99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a401a71a0 con 0x55c238caa240 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7a401a8610 con 0x55c238caf3b0 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7a401a7870 con 0x55c238ca9850 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a47bd0740 10 client.0 _ll_register_callbacks cb 0x7f7a28011c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a3e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caa240 0x55c238cb89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a3e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caa240 0x55c238cb89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36852/0 (socket says 172.21.15.77:36852) 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.693+0000 7f7a3e59c640 1 -- 172.21.15.77:0/3593361844 learned_addr learned my addr 172.21.15.77:0/3593361844 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caf3b0 0x55c238cb99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238cb8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3e59c640 1 -- 172.21.15.77:0/3593361844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 msgr2=0x55c238cb8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238cb8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3e59c640 1 -- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caf3b0 msgr2=0x55c238cb99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caf3b0 0x55c238cb99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3e59c640 1 -- 172.21.15.77:0/3593361844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c238cb9fc0 con 0x55c238caa240 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238cb8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a3e59c640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caa240 0x55c238cb89c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009730 tx=0x7f7a38009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa631bedf7fcd42 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3593361844 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3802f070 con 0x55c238caa240 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3593361844 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7a3802bc30 con 0x55c238caa240 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a28030040 con 0x55c238caa240 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3593361844 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7a3803b050 con 0x55c238caa240 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a47bd0740 10 client.15501 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c238cbb2c0 con 0x55c238caa240 2024-03-16T09:42:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3593361844 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a28030040 con 0x55c238caa240 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3593361844 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7a380035f0 con 0x55c238caa240 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a47bd0740 10 client.15501 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.694+0000 7f7a47bd0740 10 client.15501 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 20 client.15501 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238d26200 0x55c238d285f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 4 client.15501 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c238c15d20 con 0x55c238d26200 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238d26200 0x55c238d285f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a3dd9b640 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238d26200 0x55c238d285f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ce9ea948ea86d62 server_cookie=c3f536ce1ff09b0a in_seq=0 out_seq=0 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a25ffb640 10 client.15501 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a25ffb640 1 -- 172.21.15.77:0/3593361844 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c238c15d20 con 0x55c238d26200 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a25ffb640 10 client.15501 handle_command_reply: tid=0 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 client.15501 shutdown 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caa240 msgr2=0x55c238cb89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caa240 0x55c238cb89c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009730 tx=0x7f7a38009310 comp rx=0 tx=0).stop 2024-03-16T09:42:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 shutdown_connections 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55c238d26200 0x55c238d285f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c238caf3b0 0x55c238cb99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a25ffb640 0 client.15501 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c238caa240 0x55c238cb89c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 --2- 172.21.15.77:0/3593361844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c238ca9850 0x55c238cb8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 >> 172.21.15.77:0/3593361844 conn(0x55c238ca4dc0 msgr2=0x55c238ca88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 shutdown_connections 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 1 -- 172.21.15.77:0/3593361844 wait complete. 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.695+0000 7f7a47bd0740 20 client.15501 trim_cache size 0 max 16384 2024-03-16T09:42:26.706 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:42:26.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a2c0a2800 msgr2=0x7f7a2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a2c0a2800 0x7f7a2c0a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7a30006010 tx=0x7f7a30088350 comp rx=0 tx=0).stop 2024-03-16T09:42:26.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a40102290 msgr2=0x7f7a4019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a40102290 0x7f7a4019efc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a3002c630 tx=0x7f7a300328a0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 shutdown_connections 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7a2c0a2800 0x7f7a2c0a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a40102290 0x7f7a4019efc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a401018d0 0x7f7a40197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 --2- 172.21.15.77:0/3520235541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a401006e0 0x7f7a40197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 >> 172.21.15.77:0/3520235541 conn(0x7f7a400fbc70 msgr2=0x7f7a400fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 shutdown_connections 2024-03-16T09:42:26.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.696+0000 7f7a44fe6640 1 -- 172.21.15.77:0/3520235541 wait complete. 2024-03-16T09:42:26.709 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T09:42:26.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 -- 172.21.15.77:0/2326632006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc100960 msgr2=0x7f5efc100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 --2- 172.21.15.77:0/2326632006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc100960 0x7f5efc100de0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009900 tx=0x7f5eec02bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:26.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 -- 172.21.15.77:0/2326632006 shutdown_connections 2024-03-16T09:42:26.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 --2- 172.21.15.77:0/2326632006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5efc101320 0x7f5efc108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 --2- 172.21.15.77:0/2326632006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc100960 0x7f5efc100de0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 --2- 172.21.15.77:0/2326632006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc101d10 0x7f5efc102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 -- 172.21.15.77:0/2326632006 >> 172.21.15.77:0/2326632006 conn(0x7f5efc0fb410 msgr2=0x7f5efc0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 -- 172.21.15.77:0/2326632006 shutdown_connections 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 -- 172.21.15.77:0/2326632006 wait complete. 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.810+0000 7f5f035f3640 1 Processor -- start 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 -- start start 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc100960 0x7f5efc1003c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5efc101320 0x7f5efc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc101d10 0x7f5efc19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc10c220 con 0x7f5efc101d10 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc10c0a0 con 0x7f5efc100960 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc10c3a0 con 0x7f5efc101320 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc101d10 0x7f5efc19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc101d10 0x7f5efc19ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36874/0 (socket says 172.21.15.77:36874) 2024-03-16T09:42:26.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 -- 172.21.15.77:0/4223785213 learned_addr learned my addr 172.21.15.77:0/4223785213 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f0138f640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc100960 0x7f5efc1003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5efc101320 0x7f5efc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 -- 172.21.15.77:0/4223785213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5efc101320 msgr2=0x7f5efc19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5efc101320 0x7f5efc19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 -- 172.21.15.77:0/4223785213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc100960 msgr2=0x7f5efc1003c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc100960 0x7f5efc1003c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 -- 172.21.15.77:0/4223785213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5efc19f6f0 con 0x7f5efc101d10 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f01b90640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc101d10 0x7f5efc19ede0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400eb10 tx=0x7f5ef400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5ef27fc640 1 -- 172.21.15.77:0/4223785213 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4004b20 con 0x7f5efc101d10 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5ef27fc640 1 -- 172.21.15.77:0/4223785213 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef4004cc0 con 0x7f5efc101d10 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5efc19f980 con 0x7f5efc101d10 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.811+0000 7f5ef27fc640 1 -- 172.21.15.77:0/4223785213 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4010690 con 0x7f5efc101d10 2024-03-16T09:42:26.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.812+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5efc19fe50 con 0x7f5efc101d10 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5ef27fc640 1 -- 172.21.15.77:0/4223785213 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5ef4005020 con 0x7f5efc101d10 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 Processor -- start 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 -- start start 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb5850 0x555af1eb5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb6240 0x555af1ebad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ebd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555af1e1a490 con 0x555af1eb5850 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555af1dc2c90 con 0x555af1ebb3b0 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.814+0000 7f5f061d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555af1e16530 con 0x555af1eb6240 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5ef27fc640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee80a2800 0x7f5ee80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f00b8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb5850 0x555af1eb5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f0138f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb6240 0x555af1ebad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f00b8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb5850 0x555af1eb5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36886/0 (socket says 172.21.15.77:36886) 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f00b8e640 1 -- 172.21.15.77:0/1153855558 learned_addr learned my addr 172.21.15.77:0/1153855558 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ebd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f01b90640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee80a2800 0x7f5ee80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f0138f640 1 -- 172.21.15.77:0/1153855558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 msgr2=0x555af1ebd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ebd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f0138f640 1 -- 172.21.15.77:0/1153855558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb5850 msgr2=0x555af1eb5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb5850 0x555af1eb5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f0138f640 1 -- 172.21.15.77:0/1153855558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555af1e7d500 con 0x555af1eb6240 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5ef27fc640 1 -- 172.21.15.77:0/4223785213 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f5ef40cefb0 con 0x7f5efc101d10 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb6240 0x555af1ebad40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009bf0 tx=0x7f5ef8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee992cedda488dad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5f01b90640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee80a2800 0x7f5ee80a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4097dc0 tx=0x7f5ef406d5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5ee77fe640 1 -- 172.21.15.77:0/1153855558 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8004050 con 0x555af1eb6240 2024-03-16T09:42:26.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5ee77fe640 1 -- 172.21.15.77:0/1153855558 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef80054d0 con 0x555af1eb6240 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.815+0000 7f5ee77fe640 1 -- 172.21.15.77:0/1153855558 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8004d30 con 0x555af1eb6240 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 -- 172.21.15.77:0/1153855558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb6240 msgr2=0x555af1ebad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb6240 0x555af1ebad40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009bf0 tx=0x7f5ef8011810 comp rx=0 tx=0).stop 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 -- 172.21.15.77:0/1153855558 shutdown_connections 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ebd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb6240 0x555af1ebad40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1153855558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb5850 0x555af1eb5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 -- 172.21.15.77:0/1153855558 >> 172.21.15.77:0/1153855558 conn(0x555af1eb0dc0 msgr2=0x555af1eb11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 -- 172.21.15.77:0/1153855558 shutdown_connections 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 -- 172.21.15.77:0/1153855558 wait complete. 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 -- start start 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.816+0000 7f5f061d7740 1 Processor -- start 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ebfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1eb6240 0x555af1ec02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1ebb3b0 0x555af1ec72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f061d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555af1ebe360 con 0x555af1ebb3b0 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f061d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555af1dd6230 con 0x555af1eb6240 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f061d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555af1ebe4e0 con 0x555af1eb5850 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f061d7740 10 client.0 _ll_register_callbacks cb 0x555af1e12010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1ebb3b0 0x555af1ec72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1ebb3b0 0x555af1ec72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36888/0 (socket says 172.21.15.77:36888) 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 -- 172.21.15.77:0/2390292391 learned_addr learned my addr 172.21.15.77:0/2390292391 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f0138f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1eb6240 0x555af1ec02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ebfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 -- 172.21.15.77:0/2390292391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 msgr2=0x555af1ebfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ebfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 -- 172.21.15.77:0/2390292391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1eb6240 msgr2=0x555af1ec02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1eb6240 0x555af1ec02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 -- 172.21.15.77:0/2390292391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555af1de5100 con 0x555af1ebb3b0 2024-03-16T09:42:26.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1ebb3b0 0x555af1ec72a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5eec036040 tx=0x7f5eec005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f291f000cc4b3b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.817+0000 7f5ee4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/2390292391 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec0055c0 con 0x555af1ebb3b0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/2390292391 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f5eec0046c0 con 0x555af1ebb3b0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ef80054d0 con 0x555af1ebb3b0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/2390292391 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5eec0048c0 con 0x555af1ebb3b0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 10 client.15513 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555af1b2e3f0 con 0x555af1ebb3b0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/2390292391 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef80054d0 con 0x555af1ebb3b0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/2390292391 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f5eec003c20 con 0x555af1ebb3b0 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 10 client.15513 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 10 client.15513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 20 client.15513 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1ecbaa0 0x555af1ecdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 4 client.15513 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:26.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555af1e279d0 con 0x555af1ecbaa0 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.818+0000 7f5f0138f640 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1ecbaa0 0x555af1ecdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.819+0000 7f5ee4ff9640 10 client.15513 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.819+0000 7f5f0138f640 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1ecbaa0 0x555af1ecdf30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8458edabc38db0c server_cookie=69efda035be6846c in_seq=0 out_seq=0 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/2390292391 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555af1e279d0 con 0x555af1ecbaa0 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5ee4ff9640 10 client.15513 handle_command_reply: tid=0 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 client.15513 shutdown 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1ebb3b0 msgr2=0x555af1ec72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1ebb3b0 0x555af1ec72a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5eec036040 tx=0x7f5eec005d50 comp rx=0 tx=0).stop 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 shutdown_connections 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1ecbaa0 0x555af1ecdf30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1ebb3b0 0x555af1ec72a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1eb6240 0x555af1ec02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5ee4ff9640 0 client.15513 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 --2- 172.21.15.77:0/2390292391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ebfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 >> 172.21.15.77:0/2390292391 conn(0x555af1eb0dc0 msgr2=0x555af1eaffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 shutdown_connections 2024-03-16T09:42:26.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 1 -- 172.21.15.77:0/2390292391 wait complete. 2024-03-16T09:42:26.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.820+0000 7f5f061d7740 20 client.15513 trim_cache size 0 max 16384 2024-03-16T09:42:26.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 Processor -- start 2024-03-16T09:42:26.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 -- start start 2024-03-16T09:42:26.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1eaf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1eafc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc1ab540 con 0x555af1eb6240 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc1a8e10 con 0x555af1ebb3b0 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f061d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc1a6c60 con 0x555af1eb5850 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36898/0 (socket says 172.21.15.77:36898) 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 -- 172.21.15.77:0/1997659733 learned_addr learned my addr 172.21.15.77:0/1997659733 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1eafc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1eaf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 -- 172.21.15.77:0/1997659733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 msgr2=0x555af1eaf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1eaf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 -- 172.21.15.77:0/1997659733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 msgr2=0x555af1eafc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1eafc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f0138f640 1 -- 172.21.15.77:0/1997659733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555af1ebf900 con 0x555af1eb6240 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.833+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1eaf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec7170 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80099b0 tx=0x7f5ef8005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9317fcce4b4f219e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5ee6ffd640 1 -- 172.21.15.77:0/1997659733 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef801e040 con 0x555af1eb6240 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5ee6ffd640 1 -- 172.21.15.77:0/1997659733 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef801f040 con 0x555af1eb6240 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5ee6ffd640 1 -- 172.21.15.77:0/1997659733 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80035f0 con 0x555af1eb6240 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 -- 172.21.15.77:0/1997659733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 msgr2=0x555af1ec7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec7170 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80099b0 tx=0x7f5ef8005d30 comp rx=0 tx=0).stop 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 -- 172.21.15.77:0/1997659733 shutdown_connections 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1eafc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec7170 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1997659733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1eaf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 -- 172.21.15.77:0/1997659733 >> 172.21.15.77:0/1997659733 conn(0x555af1eb0dc0 msgr2=0x555af1eb7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 -- 172.21.15.77:0/1997659733 shutdown_connections 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.834+0000 7f5f061d7740 1 -- 172.21.15.77:0/1997659733 wait complete. 2024-03-16T09:42:26.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 -- start start 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 Processor -- start 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ec49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ec5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc1a72d0 con 0x555af1eb6240 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc1a8740 con 0x555af1ebb3b0 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc1a79a0 con 0x555af1eb5850 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f061d7740 10 client.0 _ll_register_callbacks cb 0x7f5eec005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36906/0 (socket says 172.21.15.77:36906) 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 -- 172.21.15.77:0/1312186688 learned_addr learned my addr 172.21.15.77:0/1312186688 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ec5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ec49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 -- 172.21.15.77:0/1312186688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 msgr2=0x555af1ec49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ec49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 -- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 msgr2=0x555af1ec5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ec5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.835+0000 7f5f0138f640 1 -- 172.21.15.77:0/1312186688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555af1bf87d0 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f0138f640 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec4f00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8005e00 tx=0x7f5ef80094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61658c9c9e5066e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5ee4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/1312186688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8004470 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/1312186688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f5ef80035f0 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ef801f040 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/1312186688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5ef80037f0 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 10 client.15519 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555af1ec7630 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/1312186688 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef801f040 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/1312186688 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f5ef803e020 con 0x555af1eb6240 2024-03-16T09:42:26.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 10 client.15519 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 10 client.15519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 20 client.15519 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1f323a0 0x555af1f34790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 4 client.15519 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555af1e21d20 con 0x555af1f323a0 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.836+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1f323a0 0x555af1f34790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5ee4ff9640 10 client.15519 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f00b8e640 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1f323a0 0x555af1f34790 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e678dc7477c7302 server_cookie=1d21be91b0462bbd in_seq=0 out_seq=0 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5ee4ff9640 1 -- 172.21.15.77:0/1312186688 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555af1e21d20 con 0x555af1f323a0 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5ee4ff9640 10 client.15519 handle_command_reply: tid=0 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 client.15519 shutdown 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 msgr2=0x555af1ec4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec4f00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8005e00 tx=0x7f5ef80094e0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 shutdown_connections 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555af1f323a0 0x555af1f34790 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555af1ebb3b0 0x555af1ec5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5ee4ff9640 0 client.15519 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555af1eb6240 0x555af1ec4f00 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 --2- 172.21.15.77:0/1312186688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555af1eb5850 0x555af1ec49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 >> 172.21.15.77:0/1312186688 conn(0x555af1eb0dc0 msgr2=0x555af1eb4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 shutdown_connections 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 1 -- 172.21.15.77:0/1312186688 wait complete. 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.837+0000 7f5f061d7740 20 client.15519 trim_cache size 0 max 16384 2024-03-16T09:42:26.847 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:42:26.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee80a2800 msgr2=0x7f5ee80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee80a2800 0x7f5ee80a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4097dc0 tx=0x7f5ef406d5e0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc101d10 msgr2=0x7f5efc19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:26.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc101d10 0x7f5efc19ede0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400eb10 tx=0x7f5ef400c2d0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 shutdown_connections 2024-03-16T09:42:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f5ee80a2800 0x7f5ee80a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5efc101d10 0x7f5efc19ede0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5efc101320 0x7f5efc19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 --2- 172.21.15.77:0/4223785213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc100960 0x7f5efc1003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 >> 172.21.15.77:0/4223785213 conn(0x7f5efc0fb410 msgr2=0x7f5efc104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 shutdown_connections 2024-03-16T09:42:26.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:26.838+0000 7f5f035f3640 1 -- 172.21.15.77:0/4223785213 wait complete. 2024-03-16T09:42:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:26 smithi077 ceph-mon[32171]: pgmap v362: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T09:42:27.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:26 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3074119161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:42:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:26 smithi119 ceph-mon[35632]: pgmap v362: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T09:42:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:26 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3074119161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:42:27.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:26 smithi129 ceph-mon[35841]: pgmap v362: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-16T09:42:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:26 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3074119161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:42:27.852 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T09:42:27.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:42:27.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 -- 172.21.15.77:0/3805791039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81018f0 msgr2=0x7f85c8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 --2- 172.21.15.77:0/3805791039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81018f0 0x7f85c8101d70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009900 tx=0x7f85b002bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 -- 172.21.15.77:0/3805791039 shutdown_connections 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 --2- 172.21.15.77:0/3805791039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c81022b0 0x7f85c8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 --2- 172.21.15.77:0/3805791039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81018f0 0x7f85c8101d70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 --2- 172.21.15.77:0/3805791039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c8100700 0x7f85c8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 -- 172.21.15.77:0/3805791039 >> 172.21.15.77:0/3805791039 conn(0x7f85c80fbc90 msgr2=0x7f85c80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 -- 172.21.15.77:0/3805791039 shutdown_connections 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 -- 172.21.15.77:0/3805791039 wait complete. 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.964+0000 7f85cde7e640 1 Processor -- start 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 -- start start 2024-03-16T09:42:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c8100700 0x7f85c8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c81018f0 0x7f85c8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81022b0 0x7f85c819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85c810c230 con 0x7f85c81022b0 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f85c810c0b0 con 0x7f85c8100700 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85c810c3b0 con 0x7f85c81018f0 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81022b0 0x7f85c819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81022b0 0x7f85c819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36926/0 (socket says 172.21.15.77:36926) 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 -- 172.21.15.77:0/2450021397 learned_addr learned my addr 172.21.15.77:0/2450021397 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c77fe640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c8100700 0x7f85c8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c81018f0 0x7f85c8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 -- 172.21.15.77:0/2450021397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c81018f0 msgr2=0x7f85c8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c81018f0 0x7f85c8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 -- 172.21.15.77:0/2450021397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c8100700 msgr2=0x7f85c8197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c8100700 0x7f85c8197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 -- 172.21.15.77:0/2450021397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c819f700 con 0x7f85c81022b0 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c81018f0 0x7f85c8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c7fff640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81022b0 0x7f85c819f090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f85c000efd0 tx=0x7f85c000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c4ff9640 1 -- 172.21.15.77:0/2450021397 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0003b70 con 0x7f85c81022b0 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c4ff9640 1 -- 172.21.15.77:0/2450021397 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c0003d10 con 0x7f85c81022b0 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c819f9f0 con 0x7f85c81022b0 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85c4ff9640 1 -- 172.21.15.77:0/2450021397 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0010980 con 0x7f85c81022b0 2024-03-16T09:42:27.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.965+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c819fef0 con 0x7f85c81022b0 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85c4ff9640 1 -- 172.21.15.77:0/2450021397 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f85c0004020 con 0x7f85c81022b0 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 Processor -- start 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 -- start start 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f879901c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f87995290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87995900 0x559f87997d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559f878f00d0 con 0x559f87995900 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559f87898c70 con 0x559f8798fda0 2024-03-16T09:42:27.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85d0a66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559f878ec170 con 0x559f87990790 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85c6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f879901c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87995900 0x559f87997d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85c77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f87995290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87995900 0x559f87997d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36936/0 (socket says 172.21.15.77:36936) 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.968+0000 7f85c6ffd640 1 -- 172.21.15.77:0/2973432490 learned_addr learned my addr 172.21.15.77:0/2973432490 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c4ff9640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85b40a2800 0x7f85b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c6ffd640 1 -- 172.21.15.77:0/2973432490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 msgr2=0x559f87995290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2973432490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f87995290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c6ffd640 1 -- 172.21.15.77:0/2973432490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 msgr2=0x559f879901c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2973432490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f879901c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c6ffd640 1 -- 172.21.15.77:0/2973432490 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f876cc7e0 con 0x559f87995900 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c7fff640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85b40a2800 0x7f85b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2973432490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87995900 0x559f87997d90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f85b002c6a0 tx=0x7f85b0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1812db8c9c05a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85c4ff9640 1 -- 172.21.15.77:0/2450021397 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f85c00cef60 con 0x7f85c81022b0 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85bdffb640 1 -- 172.21.15.77:0/2973432490 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0004040 con 0x559f87995900 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85bdffb640 1 -- 172.21.15.77:0/2973432490 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b0049070 con 0x559f87995900 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.969+0000 7f85bdffb640 1 -- 172.21.15.77:0/2973432490 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b00046c0 con 0x559f87995900 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 -- 172.21.15.77:0/2973432490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87995900 msgr2=0x559f87997d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85c7fff640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85b40a2800 0x7f85b40a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f85c0097d70 tx=0x7f85c0009f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2973432490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87995900 0x559f87997d90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f85b002c6a0 tx=0x7f85b0000bc0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 -- 172.21.15.77:0/2973432490 shutdown_connections 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2973432490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87995900 0x559f87997d90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2973432490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f87995290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2973432490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f879901c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 -- 172.21.15.77:0/2973432490 >> 172.21.15.77:0/2973432490 conn(0x559f87953c50 msgr2=0x559f87987300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 -- 172.21.15.77:0/2973432490 shutdown_connections 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 -- 172.21.15.77:0/2973432490 wait complete. 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 -- start start 2024-03-16T09:42:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.970+0000 7f85d0a66740 1 Processor -- start 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f8798fda0 0x559f8799a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f8799a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559f87998850 con 0x559f8798fda0 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559f878ac0d0 con 0x559f87995900 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559f879989d0 con 0x559f87990790 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 10 client.0 _ll_register_callbacks cb 0x559f87875760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f8798fda0 0x559f8799a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f8798fda0 0x559f8799a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36952/0 (socket says 172.21.15.77:36952) 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 -- 172.21.15.77:0/162299725 learned_addr learned my addr 172.21.15.77:0/162299725 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c77fe640 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f8799a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 -- 172.21.15.77:0/162299725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 msgr2=0x559f8799a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:27.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f8799a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 -- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 msgr2=0x559f879a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 -- 172.21.15.77:0/162299725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f8772cf70 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f8798fda0 0x559f8799a280 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f85b0005ec0 tx=0x7f85b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b3ddecb8903297 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85a6ffd640 1 -- 172.21.15.77:0/162299725 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0003a80 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85a6ffd640 1 -- 172.21.15.77:0/162299725 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f85b0003c20 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b0049070 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.971+0000 7f85a6ffd640 1 -- 172.21.15.77:0/162299725 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f85b0060050 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 10 client.15531 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559f875c4170 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85a6ffd640 1 -- 172.21.15.77:0/162299725 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b0049070 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85a6ffd640 1 -- 172.21.15.77:0/162299725 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f85b0040040 con 0x559f8798fda0 2024-03-16T09:42:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 10 client.15531 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 10 client.15531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 20 client.15531 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f879a5960 0x559f879a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 4 client.15531 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f878fd610 con 0x559f879a5960 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85c77fe640 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f879a5960 0x559f879a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85a6ffd640 10 client.15531 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.972+0000 7f85c77fe640 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f879a5960 0x559f879a7d50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e376e1bb95f9fbf server_cookie=1e6194b0b74beefb in_seq=0 out_seq=0 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85a6ffd640 1 -- 172.21.15.77:0/162299725 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f878fd610 con 0x559f879a5960 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85a6ffd640 10 client.15531 handle_command_reply: tid=0 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 client.15531 shutdown 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f8798fda0 msgr2=0x559f8799a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f8798fda0 0x559f8799a280 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f85b0005ec0 tx=0x7f85b00057f0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 shutdown_connections 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f879a5960 0x559f879a7d50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87990790 0x559f8799a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 --2- 172.21.15.77:0/162299725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f8798fda0 0x559f8799a280 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85a6ffd640 0 client.15531 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 >> 172.21.15.77:0/162299725 conn(0x559f87953c50 msgr2=0x559f879864e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 shutdown_connections 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 1 -- 172.21.15.77:0/162299725 wait complete. 2024-03-16T09:42:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.973+0000 7f85d0a66740 20 client.15531 trim_cache size 0 max 16384 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 Processor -- start 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 -- start start 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f8799a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f879a1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87995900 0x559f879ad530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85c81a9110 con 0x559f87990790 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f85c81a69e0 con 0x559f8798fda0 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85d0a66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85c81a4830 con 0x559f87995900 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f879a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f879a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36958/0 (socket says 172.21.15.77:36958) 2024-03-16T09:42:28.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 -- 172.21.15.77:0/3569683583 learned_addr learned my addr 172.21.15.77:0/3569683583 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f8799a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87995900 0x559f879ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 -- 172.21.15.77:0/3569683583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87995900 msgr2=0x559f879ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87995900 0x559f879ad530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 -- 172.21.15.77:0/3569683583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 msgr2=0x559f8799a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f8799a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 -- 172.21.15.77:0/3569683583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f87988050 con 0x559f87990790 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.991+0000 7f85c77fe640 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f879a1790 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x559f87990b90 tx=0x7f85b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e99a7e2b977546 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.992+0000 7f85bd7fa640 1 -- 172.21.15.77:0/3569683583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b801f630 con 0x559f87990790 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.992+0000 7f85bd7fa640 1 -- 172.21.15.77:0/3569683583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b801f7d0 con 0x559f87990790 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.992+0000 7f85bd7fa640 1 -- 172.21.15.77:0/3569683583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8004e00 con 0x559f87990790 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- 172.21.15.77:0/3569683583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 msgr2=0x559f879a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f879a1790 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x559f87990b90 tx=0x7f85b8005f90 comp rx=0 tx=0).stop 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- 172.21.15.77:0/3569683583 shutdown_connections 2024-03-16T09:42:28.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f87995900 0x559f879ad530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f879a1790 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 --2- 172.21.15.77:0/3569683583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f8798fda0 0x559f8799a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- 172.21.15.77:0/3569683583 >> 172.21.15.77:0/3569683583 conn(0x559f87953c50 msgr2=0x559f8798add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- 172.21.15.77:0/3569683583 shutdown_connections 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- 172.21.15.77:0/3569683583 wait complete. 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- start start 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 Processor -- start 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f8798fda0 0x559f8799f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f8799fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85c81a4ea0 con 0x559f87990790 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f85c81a6310 con 0x559f87995900 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.993+0000 7f85d0a66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85c81a5570 con 0x559f8798fda0 2024-03-16T09:42:28.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85d0a66740 10 client.0 _ll_register_callbacks cb 0x7f85b8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f8799fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f8799fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36960/0 (socket says 172.21.15.77:36960) 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 -- 172.21.15.77:0/2911612002 learned_addr learned my addr 172.21.15.77:0/2911612002 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f8798fda0 0x559f8799f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 -- 172.21.15.77:0/2911612002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f8798fda0 msgr2=0x559f8799f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f8798fda0 0x559f8799f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 -- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 msgr2=0x559f879a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 -- 172.21.15.77:0/2911612002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f879a1240 con 0x559f87990790 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f8798fda0 0x559f8799f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85c77fe640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f8799fc90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f85b80058b0 tx=0x7f85b8010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6fb44da191dd9f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:28.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85a6ffd640 1 -- 172.21.15.77:0/2911612002 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8027640 con 0x559f87990790 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85a6ffd640 1 -- 172.21.15.77:0/2911612002 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f85b80277e0 con 0x559f87990790 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b801f7d0 con 0x559f87990790 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85a6ffd640 1 -- 172.21.15.77:0/2911612002 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f85b8009c00 con 0x559f87990790 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85d0a66740 10 client.15537 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559f87a0c430 con 0x559f87990790 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.994+0000 7f85a6ffd640 1 -- 172.21.15.77:0/2911612002 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b801f7d0 con 0x559f87990790 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85a6ffd640 1 -- 172.21.15.77:0/2911612002 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f85b8030030 con 0x559f87990790 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85d0a66740 10 client.15537 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85d0a66740 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85d0a66740 20 client.15537 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f87a0cd80 0x559f87a0f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85d0a66740 4 client.15537 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559f878f7960 con 0x559f87a0cd80 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f87a0cd80 0x559f87a0f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85a6ffd640 10 client.15537 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.995+0000 7f85c6ffd640 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f87a0cd80 0x559f87a0f1a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2e4efacdcd593b3 server_cookie=8275947fe0fc7cb7 in_seq=0 out_seq=0 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85a6ffd640 1 -- 172.21.15.77:0/2911612002 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559f878f7960 con 0x559f87a0cd80 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85a6ffd640 10 client.15537 handle_command_reply: tid=0 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 client.15537 shutdown 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 msgr2=0x559f8799fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f8799fc90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f85b80058b0 tx=0x7f85b8010ef0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 shutdown_connections 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559f87a0cd80 0x559f87a0f1a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559f87995900 0x559f879a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85a6ffd640 0 client.15537 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f87990790 0x559f8799fc90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 --2- 172.21.15.77:0/2911612002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f8798fda0 0x559f8799f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.996+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 >> 172.21.15.77:0/2911612002 conn(0x559f87953c50 msgr2=0x559f879ac520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 shutdown_connections 2024-03-16T09:42:28.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85d0a66740 1 -- 172.21.15.77:0/2911612002 wait complete. 2024-03-16T09:42:28.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85d0a66740 20 client.15537 trim_cache size 0 max 16384 2024-03-16T09:42:28.006 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:42:28.006 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T09:42:28.006 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "e99d72d5-8d65-4897-808e-0946b418e8b7", 2024-03-16T09:42:28.006 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T09:42:28.006 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85b40a2800 msgr2=0x7f85b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85b40a2800 0x7f85b40a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f85c0097d70 tx=0x7f85c0009f90 comp rx=0 tx=0).stop 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81022b0 msgr2=0x7f85c819f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81022b0 0x7f85c819f090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f85c000efd0 tx=0x7f85c000c6a0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 shutdown_connections 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85b40a2800 0x7f85b40a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c81022b0 0x7f85c819f090 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c81018f0 0x7f85c8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 --2- 172.21.15.77:0/2450021397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c8100700 0x7f85c8197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 >> 172.21.15.77:0/2450021397 conn(0x7f85c80fbc90 msgr2=0x7f85c8069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 shutdown_connections 2024-03-16T09:42:28.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:27.997+0000 7f85cde7e640 1 -- 172.21.15.77:0/2450021397 wait complete. 2024-03-16T09:42:28.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:42:28.011 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:42:28.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 -- 172.21.15.77:0/3485113746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc100700 msgr2=0x7effbc100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 --2- 172.21.15.77:0/3485113746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc100700 0x7effbc100b20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7effb0009900 tx=0x7effb002bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 -- 172.21.15.77:0/3485113746 shutdown_connections 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 --2- 172.21.15.77:0/3485113746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effbc1022b0 0x7effbc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 --2- 172.21.15.77:0/3485113746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effbc1018f0 0x7effbc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 --2- 172.21.15.77:0/3485113746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc100700 0x7effbc100b20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 -- 172.21.15.77:0/3485113746 >> 172.21.15.77:0/3485113746 conn(0x7effbc0fbc90 msgr2=0x7effbc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 -- 172.21.15.77:0/3485113746 shutdown_connections 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.111+0000 7effc1a85640 1 -- 172.21.15.77:0/3485113746 wait complete. 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 Processor -- start 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 -- start start 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effbc100700 0x7effbc197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc1018f0 0x7effbc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effbc1022b0 0x7effbc19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effbc10c200 con 0x7effbc1018f0 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7effbc10c080 con 0x7effbc100700 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effc1a85640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7effbc10c380 con 0x7effbc1022b0 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc1018f0 0x7effbc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc1018f0 0x7effbc197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36978/0 (socket says 172.21.15.77:36978) 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 -- 172.21.15.77:0/905075351 learned_addr learned my addr 172.21.15.77:0/905075351 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbb7fe640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effbc100700 0x7effbc197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbbfff640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effbc1022b0 0x7effbc19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 -- 172.21.15.77:0/905075351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effbc1022b0 msgr2=0x7effbc19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effbc1022b0 0x7effbc19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 -- 172.21.15.77:0/905075351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effbc100700 msgr2=0x7effbc197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effbc100700 0x7effbc197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 -- 172.21.15.77:0/905075351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc19f6d0 con 0x7effbc1018f0 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbbfff640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effbc1022b0 0x7effbc19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:28.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbb7fe640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effbc100700 0x7effbc197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.112+0000 7effbaffd640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc1018f0 0x7effbc197ff0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7effac00e9f0 tx=0x7effac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.113+0000 7effb8ff9640 1 -- 172.21.15.77:0/905075351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac00cd80 con 0x7effbc1018f0 2024-03-16T09:42:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.113+0000 7effb8ff9640 1 -- 172.21.15.77:0/905075351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effac0108e0 con 0x7effbc1018f0 2024-03-16T09:42:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.113+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effbc19f9c0 con 0x7effbc1018f0 2024-03-16T09:42:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.113+0000 7effb8ff9640 1 -- 172.21.15.77:0/905075351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac018920 con 0x7effbc1018f0 2024-03-16T09:42:28.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.113+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effbc19ff10 con 0x7effbc1018f0 2024-03-16T09:42:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effb8ff9640 1 -- 172.21.15.77:0/905075351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7effac010a90 con 0x7effbc1018f0 2024-03-16T09:42:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 Processor -- start 2024-03-16T09:42:28.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 -- start start 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ede590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ee6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558522e43500 con 0x558522edeb60 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558522debbe0 con 0x558522ede170 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.115+0000 7effc4679740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558522e3f5a0 con 0x558522ee3cd0 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbbfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ede590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effb8ff9640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effa40a2800 0x7effa40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbbfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ede590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:40030/0 (socket says 172.21.15.77:40030) 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbbfff640 1 -- 172.21.15.77:0/713247746 learned_addr learned my addr 172.21.15.77:0/713247746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbaffd640 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ee6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbbfff640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effa40a2800 0x7effa40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effb8ff9640 1 -- 172.21.15.77:0/905075351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7effac0ceea0 con 0x7effbc1018f0 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbb7fe640 1 -- 172.21.15.77:0/713247746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 msgr2=0x558522ee6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbb7fe640 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ee6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbb7fe640 1 -- 172.21.15.77:0/713247746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 msgr2=0x558522ede590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbb7fe640 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ede590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbb7fe640 1 -- 172.21.15.77:0/713247746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558522dff4b0 con 0x558522edeb60 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbbfff640 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ede590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbb7fe640 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee3660 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7effb0036040 tx=0x7effb0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604ead8e198f8014 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effbbfff640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effa40a2800 0x7effa40a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x558522edf3c0 tx=0x7effb4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effa1ffb640 1 -- 172.21.15.77:0/713247746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0044020 con 0x558522edeb60 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effa1ffb640 1 -- 172.21.15.77:0/713247746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effb002cc20 con 0x558522edeb60 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.116+0000 7effa1ffb640 1 -- 172.21.15.77:0/713247746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0004920 con 0x558522edeb60 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 -- 172.21.15.77:0/713247746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 msgr2=0x558522ee3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee3660 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7effb0036040 tx=0x7effb0005d50 comp rx=0 tx=0).stop 2024-03-16T09:42:28.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 -- 172.21.15.77:0/713247746 shutdown_connections 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ee6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee3660 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 --2- 172.21.15.77:0/713247746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ede590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 -- 172.21.15.77:0/713247746 >> 172.21.15.77:0/713247746 conn(0x558522ed96e0 msgr2=0x558522ed9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 -- 172.21.15.77:0/713247746 shutdown_connections 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 -- 172.21.15.77:0/713247746 wait complete. 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.117+0000 7effc4679740 1 -- start start 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 1 Processor -- start 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ede170 0x558522ee85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ee3cd0 0x558522eefb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558522ee6c20 con 0x558522edeb60 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558522dff160 con 0x558522ee3cd0 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558522ee6da0 con 0x558522ede170 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effc4679740 10 client.0 _ll_register_callbacks cb 0x558522de3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:28.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36988/0 (socket says 172.21.15.77:36988) 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 -- 172.21.15.77:0/355635971 learned_addr learned my addr 172.21.15.77:0/355635971 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbb7fe640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ede170 0x558522ee85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbbfff640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ee3cd0 0x558522eefb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 -- 172.21.15.77:0/355635971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ede170 msgr2=0x558522ee85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ede170 0x558522ee85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 -- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ee3cd0 msgr2=0x558522eefb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ee3cd0 0x558522eefb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 -- 172.21.15.77:0/355635971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558522c217d0 con 0x558522edeb60 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbb7fe640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ede170 0x558522ee85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7effbaffd640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee8b10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7effac098220 tx=0x7effac010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d236e70add6304 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.118+0000 7eff9affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7eff9affd640 1 -- 172.21.15.77:0/355635971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac026040 con 0x558522edeb60 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7eff9affd640 1 -- 172.21.15.77:0/355635971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7effac0a3d60 con 0x558522edeb60 2024-03-16T09:42:28.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effb002cc20 con 0x558522edeb60 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7eff9affd640 1 -- 172.21.15.77:0/355635971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7effac09f9a0 con 0x558522edeb60 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 10 client.15549 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558522b573f0 con 0x558522edeb60 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7eff9affd640 1 -- 172.21.15.77:0/355635971 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effb002cc20 con 0x558522edeb60 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7eff9affd640 1 -- 172.21.15.77:0/355635971 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7effac09aca0 con 0x558522edeb60 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 10 client.15549 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 10 client.15549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 20 client.15549 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522ef3fd0 0x558522ef6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 4 client.15549 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558522e50a40 con 0x558522ef3fd0 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.119+0000 7effbb7fe640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522ef3fd0 0x558522ef6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7eff9affd640 10 client.15549 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effbb7fe640 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522ef3fd0 0x558522ef6460 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0cdfabce2dca8b7 server_cookie=81ab4aaf22bd7423 in_seq=0 out_seq=0 2024-03-16T09:42:28.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7eff9affd640 1 -- 172.21.15.77:0/355635971 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558522e50a40 con 0x558522ef3fd0 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7eff9affd640 10 client.15549 handle_command_reply: tid=0 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 client.15549 shutdown 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 msgr2=0x558522ee8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee8b10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7effac098220 tx=0x7effac010de0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 shutdown_connections 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522ef3fd0 0x558522ef6460 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ee3cd0 0x558522eefb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7eff9affd640 0 client.15549 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522ee8b10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 --2- 172.21.15.77:0/355635971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ede170 0x558522ee85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 >> 172.21.15.77:0/355635971 conn(0x558522ed96e0 msgr2=0x558522ed8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 shutdown_connections 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.120+0000 7effc4679740 1 -- 172.21.15.77:0/355635971 wait complete. 2024-03-16T09:42:28.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.121+0000 7effc4679740 20 client.15549 trim_cache size 0 max 16384 2024-03-16T09:42:28.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 Processor -- start 2024-03-16T09:42:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 -- start start 2024-03-16T09:42:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ee7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eefb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ed8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1ab4f0 con 0x558522edeb60 2024-03-16T09:42:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1a8dc0 con 0x558522ede170 2024-03-16T09:42:28.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effc4679740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1a6c10 con 0x558522ee3cd0 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eefb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eefb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36996/0 (socket says 172.21.15.77:36996) 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effbaffd640 1 -- 172.21.15.77:0/2785962663 learned_addr learned my addr 172.21.15.77:0/2785962663 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.137+0000 7effbb7fe640 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ee7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effbbfff640 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ed8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effbaffd640 1 -- 172.21.15.77:0/2785962663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 msgr2=0x558522ed8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effbaffd640 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ed8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effbaffd640 1 -- 172.21.15.77:0/2785962663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 msgr2=0x558522ee7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effbaffd640 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ee7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effbaffd640 1 -- 172.21.15.77:0/2785962663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558522b170f0 con 0x558522edeb60 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effbaffd640 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eefb10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7effac0975a0 tx=0x7effac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf516f11f96a50d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effa17fa640 1 -- 172.21.15.77:0/2785962663 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac0a3ff0 con 0x558522edeb60 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effa17fa640 1 -- 172.21.15.77:0/2785962663 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effac09aca0 con 0x558522edeb60 2024-03-16T09:42:28.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effa17fa640 1 -- 172.21.15.77:0/2785962663 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac09cb80 con 0x558522edeb60 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effc4679740 1 -- 172.21.15.77:0/2785962663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 msgr2=0x558522eefb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effc4679740 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eefb10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7effac0975a0 tx=0x7effac000bc0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effc4679740 1 -- 172.21.15.77:0/2785962663 shutdown_connections 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effc4679740 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522ed8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.138+0000 7effc4679740 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eefb10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 --2- 172.21.15.77:0/2785962663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522ee7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 -- 172.21.15.77:0/2785962663 >> 172.21.15.77:0/2785962663 conn(0x558522ed96e0 msgr2=0x558522ed8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 -- 172.21.15.77:0/2785962663 shutdown_connections 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 -- 172.21.15.77:0/2785962663 wait complete. 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 -- start start 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 Processor -- start 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522eed450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eed990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522eee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1a7280 con 0x558522edeb60 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1a86f0 con 0x558522ede170 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1a7950 con 0x558522ee3cd0 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effc4679740 10 client.0 _ll_register_callbacks cb 0x7effac097b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eed990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37008/0 (socket says 172.21.15.77:37008) 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effbaffd640 1 -- 172.21.15.77:0/618022681 learned_addr learned my addr 172.21.15.77:0/618022681 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effbb7fe640 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522eed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.139+0000 7effbbfff640 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522eee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effbaffd640 1 -- 172.21.15.77:0/618022681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 msgr2=0x558522eee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effbaffd640 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522eee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effbaffd640 1 -- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 msgr2=0x558522eed450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effbaffd640 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522eed450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effbaffd640 1 -- 172.21.15.77:0/618022681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558522e0a9d0 con 0x558522edeb60 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effbaffd640 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eed990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7effac0962f0 tx=0x7effac09cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4c996936a8d6d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:28.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7eff9affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7eff9affd640 1 -- 172.21.15.77:0/618022681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac0a3ff0 con 0x558522edeb60 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7eff9affd640 1 -- 172.21.15.77:0/618022681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7effac010640 con 0x558522edeb60 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effac09aca0 con 0x558522edeb60 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7eff9affd640 1 -- 172.21.15.77:0/618022681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7effac018ac0 con 0x558522edeb60 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 10 client.15555 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558522e0a9d0 con 0x558522edeb60 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7eff9affd640 1 -- 172.21.15.77:0/618022681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effac09aca0 con 0x558522edeb60 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7eff9affd640 1 -- 172.21.15.77:0/618022681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7effac0a08c0 con 0x558522edeb60 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 10 client.15555 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 10 client.15555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 20 client.15555 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522f5a8b0 0x558522f5ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 4 client.15555 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:42:28.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.140+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558522e4ad90 con 0x558522f5a8b0 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effbb7fe640 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522f5a8b0 0x558522f5ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7eff9affd640 10 client.15555 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effbb7fe640 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522f5a8b0 0x558522f5ccd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1cdbd47ad5710bd server_cookie=37f6495234b7260d in_seq=0 out_seq=0 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7eff9affd640 1 -- 172.21.15.77:0/618022681 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558522e4ad90 con 0x558522f5a8b0 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7eff9affd640 10 client.15555 handle_command_reply: tid=0 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 client.15555 shutdown 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 msgr2=0x558522eed990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eed990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7effac0962f0 tx=0x7effac09cd90 comp rx=0 tx=0).stop 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 shutdown_connections 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558522f5a8b0 0x558522f5ccd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558522ee3cd0 0x558522eee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7eff9affd640 0 client.15555 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558522edeb60 0x558522eed990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 --2- 172.21.15.77:0/618022681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558522ede170 0x558522eed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 >> 172.21.15.77:0/618022681 conn(0x558522ed96e0 msgr2=0x558522edd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 shutdown_connections 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 1 -- 172.21.15.77:0/618022681 wait complete. 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.141+0000 7effc4679740 20 client.15555 trim_cache size 0 max 16384 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout: "e99d72d5-8d65-4897-808e-0946b418e8b7": { 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "e99d72d5-8d65-4897-808e-0946b418e8b7", 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T09:42:28.152 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:42:28.153 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:42:28.153 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:42:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effa40a2800 msgr2=0x7effa40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effa40a2800 0x7effa40a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x558522edf3c0 tx=0x7effb4004ab0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc1018f0 msgr2=0x7effbc197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc1018f0 0x7effbc197ff0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7effac00e9f0 tx=0x7effac00eeb0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 shutdown_connections 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7effa40a2800 0x7effa40a4cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effbc1022b0 0x7effbc19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effbc1018f0 0x7effbc197ff0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 --2- 172.21.15.77:0/905075351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effbc100700 0x7effbc197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 >> 172.21.15.77:0/905075351 conn(0x7effbc0fbc90 msgr2=0x7effbc069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 shutdown_connections 2024-03-16T09:42:28.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:28.142+0000 7effc1a85640 1 -- 172.21.15.77:0/905075351 wait complete. 2024-03-16T09:42:28.155 INFO:tasks.cephfs.filesystem:scrub status for tag:e99d72d5-8d65-4897-808e-0946b418e8b7 - {'path': '/', 'tag': 'e99d72d5-8d65-4897-808e-0946b418e8b7', 'options': 'recursive,force'} 2024-03-16T09:42:29.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:28 smithi119 ceph-mon[35632]: pgmap v363: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:42:29.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:28 smithi119 ceph-mon[35632]: scrub queued for path: / 2024-03-16T09:42:29.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:28 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:42:29.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:28 smithi119 ceph-mon[35632]: scrub summary: active paths [/] 2024-03-16T09:42:29.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:28 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:42:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:28 smithi077 ceph-mon[32171]: pgmap v363: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:42:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:28 smithi077 ceph-mon[32171]: scrub queued for path: / 2024-03-16T09:42:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:28 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:42:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:28 smithi077 ceph-mon[32171]: scrub summary: active paths [/] 2024-03-16T09:42:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:28 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:42:29.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:28 smithi129 ceph-mon[35841]: pgmap v363: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:42:29.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:28 smithi129 ceph-mon[35841]: scrub queued for path: / 2024-03-16T09:42:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:28 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:42:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:28 smithi129 ceph-mon[35841]: scrub summary: active paths [/] 2024-03-16T09:42:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:28 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:42:31.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:30 smithi119 ceph-mon[35632]: pgmap v364: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:42:31.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:30 smithi119 ceph-mon[35632]: scrub summary: idle 2024-03-16T09:42:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:30 smithi077 ceph-mon[32171]: pgmap v364: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:42:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:30 smithi077 ceph-mon[32171]: scrub summary: idle 2024-03-16T09:42:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:30 smithi129 ceph-mon[35841]: pgmap v364: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:42:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:30 smithi129 ceph-mon[35841]: scrub summary: idle 2024-03-16T09:42:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:32 smithi119 ceph-mon[35632]: pgmap v365: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-16T09:42:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:32 smithi077 ceph-mon[32171]: pgmap v365: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-16T09:42:33.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:32 smithi129 ceph-mon[35841]: pgmap v365: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 -- 172.21.15.77:0/437772867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 msgr2=0x7f55bc102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 --2- 172.21.15.77:0/437772867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc102e90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009900 tx=0x7f55ac02bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 -- 172.21.15.77:0/437772867 shutdown_connections 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 --2- 172.21.15.77:0/437772867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55bc1033d0 0x7f55bc109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 --2- 172.21.15.77:0/437772867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc102e90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 --2- 172.21.15.77:0/437772867 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55bc101820 0x7f55bc101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 -- 172.21.15.77:0/437772867 >> 172.21.15.77:0/437772867 conn(0x7f55bc0fcd70 msgr2=0x7f55bc0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 -- 172.21.15.77:0/437772867 shutdown_connections 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 -- 172.21.15.77:0/437772867 wait complete. 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 Processor -- start 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.971+0000 7f55c3777640 1 -- start start 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55bc101820 0x7f55bc075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55bc1033d0 0x7f55bc074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10d330 con 0x7f55bc102a10 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10d1b0 con 0x7f55bc101820 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10d4b0 con 0x7f55bc1033d0 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37024/0 (socket says 172.21.15.77:37024) 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 -- 172.21.15.77:0/3616677066 learned_addr learned my addr 172.21.15.77:0/3616677066 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c1513640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55bc101820 0x7f55bc075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c1d14640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55bc1033d0 0x7f55bc074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 -- 172.21.15.77:0/3616677066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55bc1033d0 msgr2=0x7f55bc074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55bc1033d0 0x7f55bc074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 -- 172.21.15.77:0/3616677066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55bc101820 msgr2=0x7f55bc075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55bc101820 0x7f55bc075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 -- 172.21.15.77:0/3616677066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc0749a0 con 0x7f55bc102a10 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c1513640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55bc101820 0x7f55bc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c0d12640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc073bd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f55ac006010 tx=0x7f55ac005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55b27fc640 1 -- 172.21.15.77:0/3616677066 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac043020 con 0x7f55bc102a10 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55b27fc640 1 -- 172.21.15.77:0/3616677066 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55ac0043d0 con 0x7f55bc102a10 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc1a1a70 con 0x7f55bc102a10 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55b27fc640 1 -- 172.21.15.77:0/3616677066 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac003b80 con 0x7f55bc102a10 2024-03-16T09:42:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.972+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc1a1fc0 con 0x7f55bc102a10 2024-03-16T09:42:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.975+0000 7f55b27fc640 1 -- 172.21.15.77:0/3616677066 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f55ac046020 con 0x7f55bc102a10 2024-03-16T09:42:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.975+0000 7f55c6366740 1 Processor -- start 2024-03-16T09:42:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c6366740 1 -- start start 2024-03-16T09:42:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c6366740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x561846945760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c6366740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x56184694a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c6366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56184694aea0 0x56184694d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c6366740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5618468aaa40 con 0x561846945d30 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c6366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561846853920 con 0x56184694aea0 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c6366740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5618468a6ae0 con 0x561846945340 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55b27fc640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f55a80a28a0 0x7f55a80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1513640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x56184694a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1d14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x561846945760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1513640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x56184694a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37026/0 (socket says 172.21.15.77:37026) 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c0d12640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56184694aea0 0x56184694d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1d14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x561846945760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:51466/0 (socket says 172.21.15.77:51466) 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1513640 1 -- 172.21.15.77:0/174868178 learned_addr learned my addr 172.21.15.77:0/174868178 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1d14640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f55a80a28a0 0x7f55a80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55b27fc640 1 -- 172.21.15.77:0/3616677066 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f55ac0f0c60 con 0x7f55bc102a10 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c0d12640 1 -- 172.21.15.77:0/174868178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 msgr2=0x561846945760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c0d12640 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x561846945760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c0d12640 1 -- 172.21.15.77:0/174868178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 msgr2=0x56184694a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c0d12640 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x56184694a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c0d12640 1 -- 172.21.15.77:0/174868178 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56184686b460 con 0x56184694aea0 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1d14640 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x561846945760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.976+0000 7f55c1513640 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x56184694a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55c0d12640 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56184694aea0 0x56184694d330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0bd310 tx=0x7f55ac0b99e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43971c17dd0f7138 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55c1d14640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f55a80a28a0 0x7f55a80a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x561846946590 tx=0x7f55b4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55a77fe640 1 -- 172.21.15.77:0/174868178 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0bcf70 con 0x56184694aea0 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55a77fe640 1 -- 172.21.15.77:0/174868178 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55ac0bdd10 con 0x56184694aea0 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55a77fe640 1 -- 172.21.15.77:0/174868178 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0c2680 con 0x56184694aea0 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55c6366740 1 -- 172.21.15.77:0/174868178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56184694aea0 msgr2=0x56184694d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:34.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55c6366740 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56184694aea0 0x56184694d330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0bd310 tx=0x7f55ac0b99e0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55c6366740 1 -- 172.21.15.77:0/174868178 shutdown_connections 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55c6366740 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56184694aea0 0x56184694d330 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.977+0000 7f55c6366740 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x56184694a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 --2- 172.21.15.77:0/174868178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x561846945760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 -- 172.21.15.77:0/174868178 >> 172.21.15.77:0/174868178 conn(0x5618469408b0 msgr2=0x561846940cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 -- 172.21.15.77:0/174868178 shutdown_connections 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 -- 172.21.15.77:0/174868178 wait complete. 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 -- start start 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 Processor -- start 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x56184694f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x56184694f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56184694aea0 0x561846956910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56184694dc10 con 0x56184694aea0 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561846866d70 con 0x561846945d30 2024-03-16T09:42:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56184694dd90 con 0x561846945340 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c6366740 10 client.0 _ll_register_callbacks cb 0x5618465bea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c0d12640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x56184694f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c1d14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56184694aea0 0x561846956910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c1513640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x56184694f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c0d12640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x56184694f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:40100/0 (socket says 172.21.15.77:40100) 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c0d12640 1 -- 172.21.15.77:0/1386454406 learned_addr learned my addr 172.21.15.77:0/1386454406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.978+0000 7f55c1d14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56184694aea0 0x561846956910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37028/0 (socket says 172.21.15.77:37028) 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c0d12640 1 -- 172.21.15.77:0/1386454406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 msgr2=0x56184694f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c0d12640 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x56184694f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c0d12640 1 -- 172.21.15.77:0/1386454406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56184694aea0 msgr2=0x561846956910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c0d12640 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56184694aea0 0x561846956910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c0d12640 1 -- 172.21.15.77:0/1386454406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618465bf210 con 0x561846945d30 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c0d12640 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x56184694f940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0b9a70 tx=0x7f55ac0a80b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19cbf4d2d33f690 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:42:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55a4ff9640 1 -- 172.21.15.77:0/1386454406 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0bcf70 con 0x561846945d30 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55a4ff9640 1 -- 172.21.15.77:0/1386454406 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f55ac0bb910 con 0x561846945d30 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55ac0bdd10 con 0x561846945d30 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55a4ff9640 1 -- 172.21.15.77:0/1386454406 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f55ac0bd610 con 0x561846945d30 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c6366740 10 client.25457 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56184668b620 con 0x561846945d30 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.979+0000 7f55a4ff9640 1 -- 172.21.15.77:0/1386454406 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55ac0bdd10 con 0x561846945d30 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55a4ff9640 1 -- 172.21.15.77:0/1386454406 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f55ac0054b0 con 0x561846945d30 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c6366740 10 client.25457 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c6366740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c6366740 20 client.25457 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c6366740 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56184695aff0 0x56184695d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c6366740 4 client.25457 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618468b7f80 con 0x56184695aff0 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c1513640 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56184695aff0 0x56184695d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55c1513640 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56184695aff0 0x56184695d3e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9534cba1a578e5c6 server_cookie=e01c67475114b32 in_seq=0 out_seq=0 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.980+0000 7f55a4ff9640 10 client.25457 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55a4ff9640 1 -- 172.21.15.77:0/1386454406 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618468b7f80 con 0x56184695aff0 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55a4ff9640 10 client.25457 handle_command_reply: tid=0 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 client.25457 shutdown 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 msgr2=0x56184694f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x56184694f940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0b9a70 tx=0x7f55ac0a80b0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 shutdown_connections 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x56184695aff0 0x56184695d3e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56184694aea0 0x561846956910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x56184694f940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 --2- 172.21.15.77:0/1386454406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561846945340 0x56184694f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55a4ff9640 0 client.25457 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 >> 172.21.15.77:0/1386454406 conn(0x5618469408b0 msgr2=0x56184693fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 shutdown_connections 2024-03-16T09:42:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 1 -- 172.21.15.77:0/1386454406 wait complete. 2024-03-16T09:42:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.981+0000 7f55c6366740 20 client.25457 trim_cache size 0 max 16384 2024-03-16T09:42:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 Processor -- start 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 -- start start 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945340 0x56184693fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x5618469568d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846961c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1ad5e0 con 0x561846945340 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1aaeb0 con 0x561846945d30 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c6366740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1a8d00 con 0x56184694aea0 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c1513640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945340 0x56184693fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c1513640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945340 0x56184693fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37044/0 (socket says 172.21.15.77:37044) 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c1513640 1 -- 172.21.15.77:0/1381138583 learned_addr learned my addr 172.21.15.77:0/1381138583 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c0d12640 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x5618469568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.996+0000 7f55c1d14640 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846961c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55c1513640 1 -- 172.21.15.77:0/1381138583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 msgr2=0x561846961c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55c1513640 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846961c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55c1513640 1 -- 172.21.15.77:0/1381138583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 msgr2=0x5618469568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55c1513640 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x5618469568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55c1513640 1 -- 172.21.15.77:0/1381138583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56184695d750 con 0x561846945340 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55c1d14640 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846961c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55c1513640 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945340 0x56184693fa30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009fd0 tx=0x7f55b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93324ef34b206e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55a6ffd640 1 -- 172.21.15.77:0/1381138583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8011890 con 0x561846945340 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55a6ffd640 1 -- 172.21.15.77:0/1381138583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55b8005020 con 0x561846945340 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.997+0000 7f55a6ffd640 1 -- 172.21.15.77:0/1381138583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8004a00 con 0x561846945340 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- 172.21.15.77:0/1381138583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945340 msgr2=0x56184693fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945340 0x56184693fa30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009fd0 tx=0x7f55b8005f90 comp rx=0 tx=0).stop 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- 172.21.15.77:0/1381138583 shutdown_connections 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846961c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945d30 0x5618469568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 --2- 172.21.15.77:0/1381138583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945340 0x56184693fa30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- 172.21.15.77:0/1381138583 >> 172.21.15.77:0/1381138583 conn(0x5618469408b0 msgr2=0x56184693f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- 172.21.15.77:0/1381138583 shutdown_connections 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- 172.21.15.77:0/1381138583 wait complete. 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- start start 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 Processor -- start 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945340 0x561846953eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x5618469543f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846955400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1a9370 con 0x561846945d30 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1aa7e0 con 0x561846945340 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1a9a40 con 0x56184694aea0 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c6366740 10 client.0 _ll_register_callbacks cb 0x7f55b8009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c0d12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x5618469543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c0d12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x5618469543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37058/0 (socket says 172.21.15.77:37058) 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c0d12640 1 -- 172.21.15.77:0/388345474 learned_addr learned my addr 172.21.15.77:0/388345474 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.998+0000 7f55c1513640 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945340 0x561846953eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c1d14640 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846955400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c0d12640 1 -- 172.21.15.77:0/388345474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 msgr2=0x561846955400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c0d12640 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846955400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c0d12640 1 -- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945340 msgr2=0x561846953eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c0d12640 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945340 0x561846953eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c0d12640 1 -- 172.21.15.77:0/388345474 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618469559f0 con 0x561846945d30 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c0d12640 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x5618469543f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f55ac005820 tx=0x7f55ac004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ce2c7b3a5c642b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55a4ff9640 1 -- 172.21.15.77:0/388345474 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0bcf70 con 0x561846945d30 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55a4ff9640 1 -- 172.21.15.77:0/388345474 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f55ac0bb910 con 0x561846945d30 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b8005020 con 0x561846945d30 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55a4ff9640 1 -- 172.21.15.77:0/388345474 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f55ac0b98b0 con 0x561846945d30 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 10 client.15573 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5618469559f0 con 0x561846945d30 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55a4ff9640 1 -- 172.21.15.77:0/388345474 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55b8005020 con 0x561846945d30 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55a4ff9640 1 -- 172.21.15.77:0/388345474 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f55ac0a6ba0 con 0x561846945d30 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 10 client.15573 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 20 client.15573 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5618469c1720 0x5618469c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 4 client.15573 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:34.999+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56184694f1f0 con 0x5618469c1720 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.000+0000 7f55c1513640 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5618469c1720 0x5618469c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.000+0000 7f55c1513640 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5618469c1720 0x5618469c3b40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=659679dfddde498d server_cookie=59104376113a4410 in_seq=0 out_seq=0 2024-03-16T09:42:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.000+0000 7f55a4ff9640 10 client.15573 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55a4ff9640 1 -- 172.21.15.77:0/388345474 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56184694f1f0 con 0x5618469c1720 2024-03-16T09:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55a4ff9640 10 client.15573 handle_command_reply: tid=0 2024-03-16T09:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 client.15573 shutdown 2024-03-16T09:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 msgr2=0x5618469543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x5618469543f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f55ac005820 tx=0x7f55ac004580 comp rx=0 tx=0).stop 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 shutdown_connections 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5618469c1720 0x5618469c3b40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56184694aea0 0x561846955400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561846945d30 0x5618469543f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 --2- 172.21.15.77:0/388345474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561846945340 0x561846953eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55a4ff9640 0 client.15573 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 >> 172.21.15.77:0/388345474 conn(0x5618469408b0 msgr2=0x56184693fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 shutdown_connections 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 1 -- 172.21.15.77:0/388345474 wait complete. 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.003+0000 7f55c6366740 20 client.15573 trim_cache size 0 max 16384 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:42:35.010 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:42:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:42:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:42:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f55a80a28a0 msgr2=0x7f55a80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f55a80a28a0 0x7f55a80a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x561846946590 tx=0x7f55b4009500 comp rx=0 tx=0).stop 2024-03-16T09:42:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 msgr2=0x7f55bc073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc073bd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f55ac006010 tx=0x7f55ac005e40 comp rx=0 tx=0).stop 2024-03-16T09:42:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 shutdown_connections 2024-03-16T09:42:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f55a80a28a0 0x7f55a80a4d60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55bc1033d0 0x7f55bc074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55bc102a10 0x7f55bc073bd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 --2- 172.21.15.77:0/3616677066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55bc101820 0x7f55bc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 >> 172.21.15.77:0/3616677066 conn(0x7f55bc0fcd70 msgr2=0x7f55bc105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 shutdown_connections 2024-03-16T09:42:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:35.004+0000 7f55c3777640 1 -- 172.21.15.77:0/3616677066 wait complete. 2024-03-16T09:42:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:34 smithi119 ceph-mon[35632]: pgmap v366: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T09:42:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:34 smithi077 ceph-mon[32171]: pgmap v366: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T09:42:35.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:34 smithi129 ceph-mon[35841]: pgmap v366: 97 pgs: 97 active+clean; 660 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T09:42:37.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:36 smithi119 ceph-mon[35632]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T09:42:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:36 smithi077 ceph-mon[32171]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T09:42:37.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:36 smithi129 ceph-mon[35841]: pgmap v367: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T09:42:39.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:38 smithi119 ceph-mon[35632]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T09:42:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:38 smithi077 ceph-mon[32171]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T09:42:39.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:38 smithi129 ceph-mon[35841]: pgmap v368: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T09:42:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:40 smithi119 ceph-mon[35632]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:41.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:40 smithi077 ceph-mon[32171]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:41.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:40 smithi129 ceph-mon[35841]: pgmap v369: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:42 smithi119 ceph-mon[35632]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:43.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:42 smithi077 ceph-mon[32171]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:43.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:42 smithi129 ceph-mon[35841]: pgmap v370: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:44 smithi119 ceph-mon[35632]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:45.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:44 smithi077 ceph-mon[32171]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:45.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:44 smithi129 ceph-mon[35841]: pgmap v371: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:46 smithi119 ceph-mon[35632]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:46 smithi077 ceph-mon[32171]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:47.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:46 smithi129 ceph-mon[35841]: pgmap v372: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.116+0000 7eff8f95b640 1 -- 172.21.15.77:0/143158941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881018f0 msgr2=0x7eff88101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.116+0000 7eff8f95b640 1 --2- 172.21.15.77:0/143158941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881018f0 0x7eff88101d70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7eff78009900 tx=0x7eff7802bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.116+0000 7eff8f95b640 1 -- 172.21.15.77:0/143158941 shutdown_connections 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.116+0000 7eff8f95b640 1 --2- 172.21.15.77:0/143158941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff881022b0 0x7eff88108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.116+0000 7eff8f95b640 1 --2- 172.21.15.77:0/143158941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881018f0 0x7eff88101d70 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.116+0000 7eff8f95b640 1 --2- 172.21.15.77:0/143158941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 0x7eff88100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.116+0000 7eff8f95b640 1 -- 172.21.15.77:0/143158941 >> 172.21.15.77:0/143158941 conn(0x7eff880fbc70 msgr2=0x7eff880fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 -- 172.21.15.77:0/143158941 shutdown_connections 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 -- 172.21.15.77:0/143158941 wait complete. 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 Processor -- start 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 -- start start 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 0x7eff88195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff881018f0 0x7eff88195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881022b0 0x7eff8819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810b180 con 0x7eff881022b0 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810b000 con 0x7eff881018f0 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.117+0000 7eff8f95b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810b300 con 0x7eff88100700 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 0x7eff88195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8cef6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff881018f0 0x7eff88195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 0x7eff88195780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:60916/0 (socket says 172.21.15.77:60916) 2024-03-16T09:42:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 -- 172.21.15.77:0/4260176843 learned_addr learned my addr 172.21.15.77:0/4260176843 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8def8640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881022b0 0x7eff8819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 -- 172.21.15.77:0/4260176843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff881018f0 msgr2=0x7eff88195cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff881018f0 0x7eff88195cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 -- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881022b0 msgr2=0x7eff8819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881022b0 0x7eff8819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 -- 172.21.15.77:0/4260176843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8819d3a0 con 0x7eff88100700 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8cef6640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff881018f0 0x7eff88195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8def8640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881022b0 0x7eff8819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 0x7eff88195780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7eff8400f860 tx=0x7eff8400fd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff7e7fc640 1 -- 172.21.15.77:0/4260176843 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84004020 con 0x7eff88100700 2024-03-16T09:42:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8819d690 con 0x7eff88100700 2024-03-16T09:42:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff7e7fc640 1 -- 172.21.15.77:0/4260176843 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff8400ed00 con 0x7eff88100700 2024-03-16T09:42:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff7e7fc640 1 -- 172.21.15.77:0/4260176843 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84003c30 con 0x7eff88100700 2024-03-16T09:42:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.118+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8819dbb0 con 0x7eff88100700 2024-03-16T09:42:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.121+0000 7eff7e7fc640 1 -- 172.21.15.77:0/4260176843 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7eff8401b020 con 0x7eff88100700 2024-03-16T09:42:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 Processor -- start 2024-03-16T09:42:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 -- start start 2024-03-16T09:42:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdac340 0x55a77fdac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdacd30 0x55a77fdb1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a77fd11a40 con 0x55a77fdac340 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a77fcba920 con 0x55a77fdacd30 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff92623740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a77fd0dae0 con 0x55a77fdb1ea0 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff8def8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdac340 0x55a77fdac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff8def8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdac340 0x55a77fdac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46288/0 (socket says 172.21.15.77:46288) 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff8def8640 1 -- 172.21.15.77:0/3662238703 learned_addr learned my addr 172.21.15.77:0/3662238703 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdacd30 0x55a77fdb1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff7e7fc640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff740a28a0 0x7eff740a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.122+0000 7eff8cef6640 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 -- 172.21.15.77:0/3662238703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdb1ea0 msgr2=0x55a77fdb4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 -- 172.21.15.77:0/3662238703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdacd30 msgr2=0x55a77fdb1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdacd30 0x55a77fdb1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 -- 172.21.15.77:0/3662238703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77fcd2460 con 0x55a77fdac340 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff7e7fc640 1 -- 172.21.15.77:0/4260176843 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7eff840ce510 con 0x7eff88100700 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff740a28a0 0x7eff740a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdac340 0x55a77fdac760 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7eff800117e0 tx=0x7eff80011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b5d32b59a0a5e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff737fe640 1 -- 172.21.15.77:0/3662238703 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80017040 con 0x55a77fdac340 2024-03-16T09:42:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff737fe640 1 -- 172.21.15.77:0/3662238703 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff80010bc0 con 0x55a77fdac340 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff737fe640 1 -- 172.21.15.77:0/3662238703 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8001ce50 con 0x55a77fdac340 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.123+0000 7eff8def8640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff740a28a0 0x7eff740a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff80022000 tx=0x7eff80023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 -- 172.21.15.77:0/3662238703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdac340 msgr2=0x55a77fdac760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdac340 0x55a77fdac760 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7eff800117e0 tx=0x7eff80011ca0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 -- 172.21.15.77:0/3662238703 shutdown_connections 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdacd30 0x55a77fdb1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 --2- 172.21.15.77:0/3662238703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdac340 0x55a77fdac760 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 -- 172.21.15.77:0/3662238703 >> 172.21.15.77:0/3662238703 conn(0x55a77fda78b0 msgr2=0x55a77fda7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 -- 172.21.15.77:0/3662238703 shutdown_connections 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 -- 172.21.15.77:0/3662238703 wait complete. 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.124+0000 7eff92623740 1 -- start start 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 1 Processor -- start 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a77fdb4e50 con 0x55a77fdacd30 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff8d6f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff8cef6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff8d6f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:60932/0 (socket says 172.21.15.77:60932) 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff8d6f7640 1 -- 172.21.15.77:0/3964680764 learned_addr learned my addr 172.21.15.77:0/3964680764 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a77fccdd70 con 0x55a77fdb1ea0 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a77fdb4fd0 con 0x55a77fdac340 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff8def8640 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff92623740 10 client.0 _ll_register_callbacks cb 0x55a77fa25a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff8d6f7640 1 -- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 msgr2=0x55a77fdbdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.125+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff8d6f7640 1 -- 172.21.15.77:0/3964680764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 msgr2=0x55a77fdb6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff8d6f7640 1 -- 172.21.15.77:0/3964680764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77fa26210 con 0x55a77fdac340 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb6650 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7eff840cc020 tx=0x7eff84097160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a7d5948eceddf94 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff70ff9640 1 -- 172.21.15.77:0/3964680764 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8409a820 con 0x55a77fdac340 2024-03-16T09:42:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff80010bc0 con 0x55a77fdac340 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff70ff9640 1 -- 172.21.15.77:0/3964680764 <== mon.1 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7eff840991c0 con 0x55a77fdac340 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff70ff9640 1 -- 172.21.15.77:0/3964680764 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff84004350 con 0x55a77fdac340 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 10 client.25396 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a77faf2620 con 0x55a77fdac340 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff70ff9640 1 -- 172.21.15.77:0/3964680764 <== mon.1 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff80010bc0 con 0x55a77fdac340 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff70ff9640 1 -- 172.21.15.77:0/3964680764 <== mon.1 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7eff840b1030 con 0x55a77fdac340 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 10 client.25396 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 10 client.25396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 20 client.25396 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fdc2500 0x55a77fdc4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 4 client.25396 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.126+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a77fd1ef80 con 0x55a77fdc2500 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.127+0000 7eff8cef6640 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fdc2500 0x55a77fdc4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.127+0000 7eff70ff9640 10 client.25396 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.127+0000 7eff8cef6640 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fdc2500 0x55a77fdc4910 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cc12417254cd471 server_cookie=d3e846ab1d515848 in_seq=0 out_seq=0 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff70ff9640 1 -- 172.21.15.77:0/3964680764 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a77fd1ef80 con 0x55a77fdc2500 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff70ff9640 10 client.25396 handle_command_reply: tid=0 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 client.25396 shutdown 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 msgr2=0x55a77fdb6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb6650 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7eff840cc020 tx=0x7eff84097160 comp rx=0 tx=0).stop 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 shutdown_connections 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fdc2500 0x55a77fdc4910 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff70ff9640 0 client.25396 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 --2- 172.21.15.77:0/3964680764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb6650 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 >> 172.21.15.77:0/3964680764 conn(0x55a77fda78b0 msgr2=0x55a77fda6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 shutdown_connections 2024-03-16T09:42:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 1 -- 172.21.15.77:0/3964680764 wait complete. 2024-03-16T09:42:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.128+0000 7eff92623740 20 client.25396 trim_cache size 0 max 16384 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 Processor -- start 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 -- start start 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff881a9240 con 0x55a77fdacd30 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff881a6b10 con 0x55a77fdb1ea0 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff92623740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff881a4960 con 0x55a77fdac340 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46296/0 (socket says 172.21.15.77:46296) 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8d6f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 -- 172.21.15.77:0/33085482 learned_addr learned my addr 172.21.15.77:0/33085482 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8def8640 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 -- 172.21.15.77:0/33085482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 msgr2=0x55a77fdb5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 -- 172.21.15.77:0/33085482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 msgr2=0x55a77fdb63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8cef6640 1 -- 172.21.15.77:0/33085482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77fdc0e60 con 0x55a77fdacd30 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.143+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.144+0000 7eff8cef6640 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb5e80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7eff78000c00 tx=0x7eff7802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7c6cd79baa6fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.144+0000 7eff72ffd640 1 -- 172.21.15.77:0/33085482 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7804c020 con 0x55a77fdacd30 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.144+0000 7eff72ffd640 1 -- 172.21.15.77:0/33085482 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff78048b90 con 0x55a77fdacd30 2024-03-16T09:42:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.144+0000 7eff72ffd640 1 -- 172.21.15.77:0/33085482 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78005040 con 0x55a77fdacd30 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.144+0000 7eff92623740 1 -- 172.21.15.77:0/33085482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 msgr2=0x55a77fdb5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.144+0000 7eff92623740 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb5e80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7eff78000c00 tx=0x7eff7802c7e0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 -- 172.21.15.77:0/33085482 shutdown_connections 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdb63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdb5e80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 --2- 172.21.15.77:0/33085482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdb5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 -- 172.21.15.77:0/33085482 >> 172.21.15.77:0/33085482 conn(0x55a77fda78b0 msgr2=0x55a77fda2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 -- 172.21.15.77:0/33085482 shutdown_connections 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 -- 172.21.15.77:0/33085482 wait complete. 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 -- start start 2024-03-16T09:42:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 Processor -- start 2024-03-16T09:42:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdbb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.145+0000 7eff92623740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdbb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff92623740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdbb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdbb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46298/0 (socket says 172.21.15.77:46298) 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 -- 172.21.15.77:0/2533502133 learned_addr learned my addr 172.21.15.77:0/2533502133 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdbb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff881a4fd0 con 0x55a77fdacd30 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff881a6440 con 0x55a77fdb1ea0 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff881a56a0 con 0x55a77fdac340 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8def8640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 -- 172.21.15.77:0/2533502133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 msgr2=0x55a77fdbb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdbb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 -- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 msgr2=0x55a77fdbc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 -- 172.21.15.77:0/2533502133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77fdbcf90 con 0x55a77fdacd30 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8def8640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdbb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff8cef6640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdbb9e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7eff78006010 tx=0x7eff78048460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f780e2a4ff3fc2b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff70ff9640 1 -- 172.21.15.77:0/2533502133 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7804c030 con 0x55a77fdacd30 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.146+0000 7eff70ff9640 1 -- 172.21.15.77:0/2533502133 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7eff78005040 con 0x55a77fdacd30 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff70ff9640 10 client.0 inactive, discarding osd_map(88..88 src has 1..88) 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff92623740 10 client.0 _ll_register_callbacks cb 0x7eff8400ff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff78048b90 con 0x55a77fdacd30 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff70ff9640 1 -- 172.21.15.77:0/2533502133 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff78005240 con 0x55a77fdacd30 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff92623740 10 client.15591 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55a77fe28b70 con 0x55a77fdacd30 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff70ff9640 1 -- 172.21.15.77:0/2533502133 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7eff780053e0 con 0x55a77fdacd30 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.147+0000 7eff92623740 10 client.15591 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff92623740 10 client.15591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff92623740 20 client.15591 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff92623740 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fe293b0 0x55a77fe2b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff92623740 4 client.15591 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7eff78048b90 con 0x55a77fe293b0 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fe293b0 0x55a77fe2b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff70ff9640 10 client.15591 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.148+0000 7eff8d6f7640 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fe293b0 0x55a77fe2b7d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a344488b22efbab server_cookie=5750746ee8d7d53f in_seq=0 out_seq=0 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff70ff9640 1 -- 172.21.15.77:0/2533502133 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7eff78048b90 con 0x55a77fe293b0 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff70ff9640 10 client.15591 handle_command_reply: tid=0 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 client.15591 shutdown 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 msgr2=0x55a77fdbb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdbb9e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7eff78006010 tx=0x7eff78048460 comp rx=0 tx=0).stop 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 shutdown_connections 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a77fe293b0 0x55a77fe2b7d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff70ff9640 0 client.15591 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a77fdb1ea0 0x55a77fdbc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a77fdacd30 0x55a77fdbb9e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 --2- 172.21.15.77:0/2533502133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a77fdac340 0x55a77fdbb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.151+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 >> 172.21.15.77:0/2533502133 conn(0x55a77fda78b0 msgr2=0x55a77fda6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 shutdown_connections 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff92623740 1 -- 172.21.15.77:0/2533502133 wait complete. 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff92623740 20 client.15591 trim_cache size 0 max 16384 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:42:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:42:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff740a28a0 msgr2=0x7eff740a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff8f95b640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff740a28a0 0x7eff740a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff80022000 tx=0x7eff80023000 comp rx=0 tx=0).stop 2024-03-16T09:42:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 msgr2=0x7eff88195780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff8f95b640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 0x7eff88195780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7eff8400f860 tx=0x7eff8400fd20 comp rx=0 tx=0).stop 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.152+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 shutdown_connections 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.153+0000 7eff8f95b640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7eff740a28a0 0x7eff740a4d60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.153+0000 7eff8f95b640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881022b0 0x7eff8819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.153+0000 7eff8f95b640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff881018f0 0x7eff88195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.153+0000 7eff8f95b640 1 --2- 172.21.15.77:0/4260176843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88100700 0x7eff88195780 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.153+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 >> 172.21.15.77:0/4260176843 conn(0x7eff880fbc70 msgr2=0x7eff88104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.153+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 shutdown_connections 2024-03-16T09:42:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:42:48.153+0000 7eff8f95b640 1 -- 172.21.15.77:0/4260176843 wait complete. 2024-03-16T09:42:49.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:48 smithi119 ceph-mon[35632]: pgmap v373: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T09:42:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:48 smithi077 ceph-mon[32171]: pgmap v373: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T09:42:49.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:48 smithi129 ceph-mon[35841]: pgmap v373: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-16T09:42:51.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:50 smithi119 ceph-mon[35632]: pgmap v374: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-16T09:42:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:50 smithi077 ceph-mon[32171]: pgmap v374: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-16T09:42:51.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:50 smithi129 ceph-mon[35841]: pgmap v374: 97 pgs: 97 active+clean; 661 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-16T09:42:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:52 smithi119 ceph-mon[35632]: pgmap v375: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:52 smithi077 ceph-mon[32171]: pgmap v375: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:52 smithi129 ceph-mon[35841]: pgmap v375: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:54 smithi119 ceph-mon[35632]: pgmap v376: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:42:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:54 smithi077 ceph-mon[32171]: pgmap v376: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:42:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:54 smithi129 ceph-mon[35841]: pgmap v376: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:42:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:56 smithi119 ceph-mon[35632]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:56 smithi077 ceph-mon[32171]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:57.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:56 smithi129 ceph-mon[35841]: pgmap v377: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:42:58.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:42:58.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:42:58.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 -- 172.21.15.77:0/923961629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc102a70 msgr2=0x7f57bc102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 --2- 172.21.15.77:0/923961629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc102a70 0x7f57bc102ef0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009930 tx=0x7f57b002bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:58.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 -- 172.21.15.77:0/923961629 shutdown_connections 2024-03-16T09:42:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 --2- 172.21.15.77:0/923961629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57bc103430 0x7f57bc109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 --2- 172.21.15.77:0/923961629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc102a70 0x7f57bc102ef0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 --2- 172.21.15.77:0/923961629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 0x7f57bc0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 -- 172.21.15.77:0/923961629 >> 172.21.15.77:0/923961629 conn(0x7f57bc0fb3d0 msgr2=0x7f57bc0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 -- 172.21.15.77:0/923961629 shutdown_connections 2024-03-16T09:42:58.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.266+0000 7f57c13f4640 1 -- 172.21.15.77:0/923961629 wait complete. 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 Processor -- start 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 -- start start 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 0x7f57bc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57bc102a70 0x7f57bc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc103430 0x7f57bc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc10d310 con 0x7f57bc103430 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc10d190 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57c13f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc10d490 con 0x7f57bc102a70 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57baffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 0x7f57bc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57baffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 0x7f57bc197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50144/0 (socket says 172.21.15.77:50144) 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57baffd640 1 -- 172.21.15.77:0/2286499312 learned_addr learned my addr 172.21.15.77:0/2286499312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.267+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc103430 0x7f57bc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57bc102a70 0x7f57bc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57baffd640 1 -- 172.21.15.77:0/2286499312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57bc102a70 msgr2=0x7f57bc197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57baffd640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57bc102a70 0x7f57bc197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57baffd640 1 -- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc103430 msgr2=0x7f57bc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57baffd640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc103430 0x7f57bc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57baffd640 1 -- 172.21.15.77:0/2286499312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57bc19f670 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57baffd640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 0x7f57bc197a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00e9c0 tx=0x7f57ac00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57a3fff640 1 -- 172.21.15.77:0/2286499312 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac00ccc0 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57a3fff640 1 -- 172.21.15.77:0/2286499312 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57ac0035f0 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57bc19f960 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57a3fff640 1 -- 172.21.15.77:0/2286499312 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac00ce60 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.268+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57bc19fe80 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 Processor -- start 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 -- start start 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1f7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca1fa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589ca153500 con 0x5589ca1f2580 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5589ca0fbbe0 con 0x5589ca1f2f70 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5589ca14f5a0 con 0x5589ca1f80e0 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43120/0 (socket says 172.21.15.77:43120) 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57bb7fe640 1 -- 172.21.15.77:0/4267154727 learned_addr learned my addr 172.21.15.77:0/4267154727 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.271+0000 7f57a3fff640 1 -- 172.21.15.77:0/2286499312 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f57ac0037a0 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca1fa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57bb7fe640 1 -- 172.21.15.77:0/4267154727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 msgr2=0x5589ca1fa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca1fa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57baffd640 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1f7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57bb7fe640 1 -- 172.21.15.77:0/4267154727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 msgr2=0x5589ca1f7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1f7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57bb7fe640 1 -- 172.21.15.77:0/4267154727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ca10f4b0 con 0x5589ca1f2580 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca1fa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57baffd640 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1f7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1f29a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f57b4011440 tx=0x7f57b4011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f9c3ebd776d6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57a3fff640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f57a40a2800 0x7f57a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57a17fa640 1 -- 172.21.15.77:0/4267154727 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4003c30 con 0x5589ca1f2580 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57a17fa640 1 -- 172.21.15.77:0/4267154727 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b4003dd0 con 0x5589ca1f2580 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.272+0000 7f57a3fff640 1 -- 172.21.15.77:0/2286499312 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57ac0cef10 con 0x7f57bc0ffaa0 2024-03-16T09:42:58.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.273+0000 7f57baffd640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f57a40a2800 0x7f57a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.273+0000 7f57a17fa640 1 -- 172.21.15.77:0/4267154727 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b400fa10 con 0x5589ca1f2580 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 -- 172.21.15.77:0/4267154727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 msgr2=0x5589ca1f29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57baffd640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f57a40a2800 0x7f57a40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f57ac097cc0 tx=0x7f57ac003e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1f29a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f57b4011440 tx=0x7f57b4011900 comp rx=0 tx=0).stop 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 -- 172.21.15.77:0/4267154727 shutdown_connections 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca1fa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1f7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/4267154727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1f29a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 -- 172.21.15.77:0/4267154727 >> 172.21.15.77:0/4267154727 conn(0x5589ca1e96e0 msgr2=0x5589ca1e9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 -- 172.21.15.77:0/4267154727 shutdown_connections 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 -- 172.21.15.77:0/4267154727 wait complete. 2024-03-16T09:42:58.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 -- start start 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.274+0000 7f57c3fe4740 1 Processor -- start 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1fcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1fd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca204070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589ca1fb0e0 con 0x5589ca1f2580 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5589ca10f160 con 0x5589ca1f2f70 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5589ca1fb260 con 0x5589ca1f80e0 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57c3fe4740 10 client.0 _ll_register_callbacks cb 0x5589ca0f3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1fcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1fcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43124/0 (socket says 172.21.15.77:43124) 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57bb7fe640 1 -- 172.21.15.77:0/705389887 learned_addr learned my addr 172.21.15.77:0/705389887 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57ba7fc640 1 -- 172.21.15.77:0/705389887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 msgr2=0x5589ca204070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T09:42:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca204070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57ba7fc640 1 -- 172.21.15.77:0/705389887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 msgr2=0x5589ca1fcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1fcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57ba7fc640 1 -- 172.21.15.77:0/705389887 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589c9f317d0 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.275+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1fd030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009230 tx=0x7f57b0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f699e6d20b2510 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f579eff5640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f579eff5640 1 -- 172.21.15.77:0/705389887 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b0005d50 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b4003dd0 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f579eff5640 1 -- 172.21.15.77:0/705389887 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57b0005230 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f579eff5640 1 -- 172.21.15.77:0/705389887 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f57b0005430 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 10 client.25490 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5589c9e673f0 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f579eff5640 1 -- 172.21.15.77:0/705389887 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b4003dd0 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f579eff5640 1 -- 172.21.15.77:0/705389887 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f57b0003960 con 0x5589ca1f2f70 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 10 client.25490 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 10 client.25490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 20 client.25490 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca1ef4d0 0x5589ca1f1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 4 client.25490 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.276+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589ca160a40 con 0x5589ca1ef4d0 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.277+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca1ef4d0 0x5589ca1f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.277+0000 7f579eff5640 10 client.25490 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.277+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca1ef4d0 0x5589ca1f1910 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8294269a8e9689a5 server_cookie=b02f42079c75ae94 in_seq=0 out_seq=0 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f579eff5640 1 -- 172.21.15.77:0/705389887 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5589ca160a40 con 0x5589ca1ef4d0 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f579eff5640 10 client.25490 handle_command_reply: tid=0 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 client.25490 shutdown 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 msgr2=0x5589ca1fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1fd030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009230 tx=0x7f57b0004270 comp rx=0 tx=0).stop 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 shutdown_connections 2024-03-16T09:42:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca1ef4d0 0x5589ca1f1910 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca204070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f579eff5640 0 client.25490 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca1fd030 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/705389887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2580 0x5589ca1fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 >> 172.21.15.77:0/705389887 conn(0x5589ca1e96e0 msgr2=0x5589ca1e8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 shutdown_connections 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 1 -- 172.21.15.77:0/705389887 wait complete. 2024-03-16T09:42:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.278+0000 7f57c3fe4740 20 client.25490 trim_cache size 0 max 16384 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.310+0000 7f57c3fe4740 1 Processor -- start 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57c3fe4740 1 -- start start 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f2580 0x5589ca1ed050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca204030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f80e0 0x5589ca20ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1a9340 con 0x5589ca1f80e0 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1a6c10 con 0x5589ca1f2f70 2024-03-16T09:42:58.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1a4a60 con 0x5589ca1f2580 2024-03-16T09:42:58.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f2580 0x5589ca1ed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f2580 0x5589ca1ed050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37264/0 (socket says 172.21.15.77:37264) 2024-03-16T09:42:58.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 -- 172.21.15.77:0/1852644855 learned_addr learned my addr 172.21.15.77:0/1852644855 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f80e0 0x5589ca20ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca204030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 -- 172.21.15.77:0/1852644855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 msgr2=0x5589ca204030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca204030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 -- 172.21.15.77:0/1852644855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f80e0 msgr2=0x5589ca20ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f80e0 0x5589ca20ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.311+0000 7f57bb7fe640 1 -- 172.21.15.77:0/1852644855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589c9e270f0 con 0x5589ca1f2580 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.312+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f2580 0x5589ca1ed050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005ff0 tx=0x7f57b4036040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7138dac2d29a678 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.312+0000 7f57a0ff9640 1 -- 172.21.15.77:0/1852644855 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b403ba30 con 0x5589ca1f2580 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.312+0000 7f57a0ff9640 1 -- 172.21.15.77:0/1852644855 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b403b6e0 con 0x5589ca1f2580 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.312+0000 7f57a0ff9640 1 -- 172.21.15.77:0/1852644855 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4004260 con 0x5589ca1f2580 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 -- 172.21.15.77:0/1852644855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f2580 msgr2=0x5589ca1ed050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f2580 0x5589ca1ed050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005ff0 tx=0x7f57b4036040 comp rx=0 tx=0).stop 2024-03-16T09:42:58.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 -- 172.21.15.77:0/1852644855 shutdown_connections 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f80e0 0x5589ca20ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2f70 0x5589ca204030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/1852644855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f2580 0x5589ca1ed050 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 -- 172.21.15.77:0/1852644855 >> 172.21.15.77:0/1852644855 conn(0x5589ca1e96e0 msgr2=0x5589ca1ec940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 -- 172.21.15.77:0/1852644855 shutdown_connections 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 -- 172.21.15.77:0/1852644855 wait complete. 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 -- start start 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.313+0000 7f57c3fe4740 1 Processor -- start 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2580 0x5589ca201c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2f70 0x5589ca202160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57c3fe4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca203170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1a50d0 con 0x5589ca1f2f70 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1a6540 con 0x5589ca1f2580 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57c3fe4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1a57a0 con 0x5589ca1f80e0 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57c3fe4740 10 client.0 _ll_register_callbacks cb 0x7f57b40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:58.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2580 0x5589ca201c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57ba7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2f70 0x5589ca202160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2580 0x5589ca201c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50160/0 (socket says 172.21.15.77:50160) 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 -- 172.21.15.77:0/2665383231 learned_addr learned my addr 172.21.15.77:0/2665383231 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca203170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 -- 172.21.15.77:0/2665383231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 msgr2=0x5589ca203170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca203170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 -- 172.21.15.77:0/2665383231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2f70 msgr2=0x5589ca202160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2f70 0x5589ca202160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 -- 172.21.15.77:0/2665383231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ca11a9d0 con 0x5589ca1f2580 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2f70 0x5589ca202160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f57bb7fe640 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2580 0x5589ca201c20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009ce0 tx=0x7f57b4009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13b7edb0d1cc9e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.314+0000 7f579eff5640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:42:58.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.315+0000 7f579eff5640 1 -- 172.21.15.77:0/2665383231 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b403b440 con 0x5589ca1f2580 2024-03-16T09:42:58.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.315+0000 7f579eff5640 1 -- 172.21.15.77:0/2665383231 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57b4004000 con 0x5589ca1f2580 2024-03-16T09:42:58.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.315+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b403b6e0 con 0x5589ca1f2580 2024-03-16T09:42:58.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.315+0000 7f579eff5640 1 -- 172.21.15.77:0/2665383231 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f57b4004200 con 0x5589ca1f2580 2024-03-16T09:42:58.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.315+0000 7f57c3fe4740 10 client.25496 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.315+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5589ca26a700 con 0x5589ca1f2580 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.315+0000 7f579eff5640 1 -- 172.21.15.77:0/2665383231 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b403b6e0 con 0x5589ca1f2580 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f579eff5640 1 -- 172.21.15.77:0/2665383231 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f57b4054080 con 0x5589ca1f2580 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57c3fe4740 10 client.25496 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57c3fe4740 10 client.25496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57c3fe4740 20 client.25496 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca26ada0 0x5589ca26d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57c3fe4740 4 client.25496 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5589ca15ad90 con 0x5589ca26ada0 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca26ada0 0x5589ca26d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57ba7fc640 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca26ada0 0x5589ca26d190 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f20faae9fdb4b4 server_cookie=63c778c0ea7517bc in_seq=0 out_seq=0 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f579eff5640 10 client.25496 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:58.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f579eff5640 1 -- 172.21.15.77:0/2665383231 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5589ca15ad90 con 0x5589ca26ada0 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f579eff5640 10 client.25496 handle_command_reply: tid=0 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.316+0000 7f57c3fe4740 1 client.25496 shutdown 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2580 msgr2=0x5589ca201c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2580 0x5589ca201c20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009ce0 tx=0x7f57b4009f60 comp rx=0 tx=0).stop 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 shutdown_connections 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5589ca26ada0 0x5589ca26d190 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589ca1f80e0 0x5589ca203170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589ca1f2f70 0x5589ca202160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f579eff5640 0 client.25496 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 --2- 172.21.15.77:0/2665383231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5589ca1f2580 0x5589ca201c20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 >> 172.21.15.77:0/2665383231 conn(0x5589ca1e96e0 msgr2=0x5589ca1ed1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 shutdown_connections 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 1 -- 172.21.15.77:0/2665383231 wait complete. 2024-03-16T09:42:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c3fe4740 20 client.25496 trim_cache size 0 max 16384 2024-03-16T09:42:58.325 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:42:58.325 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T09:42:58.325 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T09:42:58.325 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:42:58.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f57a40a2800 msgr2=0x7f57a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f57a40a2800 0x7f57a40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f57ac097cc0 tx=0x7f57ac003e30 comp rx=0 tx=0).stop 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 msgr2=0x7f57bc197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 0x7f57bc197a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00e9c0 tx=0x7f57ac00edc0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 shutdown_connections 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f57a40a2800 0x7f57a40a4cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57bc103430 0x7f57bc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57bc102a70 0x7f57bc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 --2- 172.21.15.77:0/2286499312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57bc0ffaa0 0x7f57bc197a50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.317+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 >> 172.21.15.77:0/2286499312 conn(0x7f57bc0fb3d0 msgr2=0x7f57bc105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.318+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 shutdown_connections 2024-03-16T09:42:58.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.318+0000 7f57c13f4640 1 -- 172.21.15.77:0/2286499312 wait complete. 2024-03-16T09:42:58.332 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T09:42:58.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:42:58.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/65801503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a4100700 msgr2=0x7fa9a4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/65801503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a4100700 0x7fa9a4100b20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa994009900 tx=0x7fa99402bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/65801503 shutdown_connections 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/65801503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9a41022b0 0x7fa9a4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/65801503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a41018f0 0x7fa9a4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/65801503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a4100700 0x7fa9a4100b20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/65801503 >> 172.21.15.77:0/65801503 conn(0x7fa9a40fbc70 msgr2=0x7fa9a40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/65801503 shutdown_connections 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.437+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/65801503 wait complete. 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 Processor -- start 2024-03-16T09:42:58.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 -- start start 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a4100700 0x7fa9a4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9a41018f0 0x7fa9a4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a41022b0 0x7fa9a419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410c270 con 0x7fa9a41018f0 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410c0f0 con 0x7fa9a4100700 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.438+0000 7fa9aa8fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410c3f0 con 0x7fa9a41022b0 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9a41018f0 0x7fa9a4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a4100700 0x7fa9a4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9a41018f0 0x7fa9a4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43150/0 (socket says 172.21.15.77:43150) 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a4100700 0x7fa9a4197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50178/0 (socket says 172.21.15.77:50178) 2024-03-16T09:42:58.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a37fe640 1 -- 172.21.15.77:0/2881065172 learned_addr learned my addr 172.21.15.77:0/2881065172 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a8e9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a41022b0 0x7fa9a419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 -- 172.21.15.77:0/2881065172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a41022b0 msgr2=0x7fa9a419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a41022b0 0x7fa9a419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 -- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9a41018f0 msgr2=0x7fa9a4198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9a41018f0 0x7fa9a4198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 -- 172.21.15.77:0/2881065172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a419f6e0 con 0x7fa9a4100700 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a3fff640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a4100700 0x7fa9a4197ac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa994036040 tx=0x7fa9940042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a17fa640 1 -- 172.21.15.77:0/2881065172 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994043020 con 0x7fa9a4100700 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a17fa640 1 -- 172.21.15.77:0/2881065172 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa994004a50 con 0x7fa9a4100700 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9a17fa640 1 -- 172.21.15.77:0/2881065172 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994003b50 con 0x7fa9a4100700 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a4102b10 con 0x7fa9a4100700 2024-03-16T09:42:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.439+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a4103080 con 0x7fa9a4100700 2024-03-16T09:42:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.441+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a41033a0 con 0x7fa9a4100700 2024-03-16T09:42:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.442+0000 7fa9a17fa640 1 -- 172.21.15.77:0/2881065172 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa994046020 con 0x7fa9a4100700 2024-03-16T09:42:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.442+0000 7fa9a17fa640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa98c0a2880 0x7fa98c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.442+0000 7fa9a37fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa98c0a2880 0x7fa98c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.442+0000 7fa9a17fa640 1 -- 172.21.15.77:0/2881065172 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fa9940f0ad0 con 0x7fa9a4100700 2024-03-16T09:42:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.443+0000 7fa9a37fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa98c0a2880 0x7fa98c0a4d40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4198fd0 tx=0x7fa998005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.451+0000 7fa9a17fa640 1 -- 172.21.15.77:0/2881065172 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa99404d020 con 0x7fa9a4100700 2024-03-16T09:42:58.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.685+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9a4103bc0 con 0x7fa9a4100700 2024-03-16T09:42:58.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.686+0000 7fa9a17fa640 1 -- 172.21.15.77:0/2881065172 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7fa9940bdb80 con 0x7fa9a4100700 2024-03-16T09:42:58.689 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:42:58.689 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:42:58.689 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:42:58.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa98c0a2880 msgr2=0x7fa98c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa98c0a2880 0x7fa98c0a4d40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4198fd0 tx=0x7fa998005f90 comp rx=0 tx=0).stop 2024-03-16T09:42:58.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a4100700 msgr2=0x7fa9a4197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a4100700 0x7fa9a4197ac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa994036040 tx=0x7fa9940042f0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 shutdown_connections 2024-03-16T09:42:58.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa98c0a2880 0x7fa98c0a4d40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9a41022b0 0x7fa9a419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9a41018f0 0x7fa9a4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 --2- 172.21.15.77:0/2881065172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9a4100700 0x7fa9a4197ac0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 >> 172.21.15.77:0/2881065172 conn(0x7fa9a40fbc70 msgr2=0x7fa9a40fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 shutdown_connections 2024-03-16T09:42:58.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.688+0000 7fa9aa8fe640 1 -- 172.21.15.77:0/2881065172 wait complete. 2024-03-16T09:42:58.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T09:42:58.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 -- 172.21.15.77:0/1499966888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b0102a00 msgr2=0x7f23b0102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 --2- 172.21.15.77:0/1499966888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b0102a00 0x7f23b0102e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f239c009900 tx=0x7f239c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:58.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 -- 172.21.15.77:0/1499966888 shutdown_connections 2024-03-16T09:42:58.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 --2- 172.21.15.77:0/1499966888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b01033c0 0x7f23b0109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 --2- 172.21.15.77:0/1499966888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b0102a00 0x7f23b0102e80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 --2- 172.21.15.77:0/1499966888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0101810 0x7f23b0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 -- 172.21.15.77:0/1499966888 >> 172.21.15.77:0/1499966888 conn(0x7f23b00fcd80 msgr2=0x7f23b00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 -- 172.21.15.77:0/1499966888 shutdown_connections 2024-03-16T09:42:58.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.806+0000 7f23b710a640 1 -- 172.21.15.77:0/1499966888 wait complete. 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 Processor -- start 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 -- start start 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b0101810 0x7f23b0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0102a00 0x7f23b0073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b01033c0 0x7f23b0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f23b010d320 con 0x7f23b0102a00 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f23b010d1a0 con 0x7f23b01033c0 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b710a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f23b010d4a0 con 0x7f23b0101810 2024-03-16T09:42:58.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0102a00 0x7f23b0073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0102a00 0x7f23b0073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43168/0 (socket says 172.21.15.77:43168) 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23affff640 1 -- 172.21.15.77:0/333910922 learned_addr learned my addr 172.21.15.77:0/333910922 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b0101810 0x7f23b0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.807+0000 7f23b56a7640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b01033c0 0x7f23b0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23affff640 1 -- 172.21.15.77:0/333910922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b0101810 msgr2=0x7f23b0075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23affff640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b0101810 0x7f23b0075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23affff640 1 -- 172.21.15.77:0/333910922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b01033c0 msgr2=0x7f23b0074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23affff640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b01033c0 0x7f23b0074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23affff640 1 -- 172.21.15.77:0/333910922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b00749a0 con 0x7f23b0102a00 2024-03-16T09:42:58.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b0101810 0x7f23b0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23affff640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0102a00 0x7f23b0073bd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f239c006010 tx=0x7f239c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23adffb640 1 -- 172.21.15.77:0/333910922 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c043020 con 0x7f23b0102a00 2024-03-16T09:42:58.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b01a1b70 con 0x7f23b0102a00 2024-03-16T09:42:58.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.808+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b01a2110 con 0x7f23b0102a00 2024-03-16T09:42:58.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.809+0000 7f23adffb640 1 -- 172.21.15.77:0/333910922 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f239c0043d0 con 0x7f23b0102a00 2024-03-16T09:42:58.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.809+0000 7f23adffb640 1 -- 172.21.15.77:0/333910922 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c003b10 con 0x7f23b0102a00 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.811+0000 7f23adffb640 1 -- 172.21.15.77:0/333910922 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f239c046020 con 0x7f23b0102a00 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 Processor -- start 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 -- start start 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e55480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55a50 0x55a860e5a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e5abc0 0x55a860e5d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a860db1490 con 0x55a860e55a50 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a860d59c90 con 0x55a860e5abc0 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a860dad530 con 0x55a860e55060 2024-03-16T09:42:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b56a7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e55480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b56a7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e55480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37302/0 (socket says 172.21.15.77:37302) 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b56a7640 1 -- 172.21.15.77:0/601852833 learned_addr learned my addr 172.21.15.77:0/601852833 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23affff640 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e5abc0 0x55a860e5d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55a50 0x55a860e5a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23adffb640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f23a00a2880 0x7f23a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b56a7640 1 -- 172.21.15.77:0/601852833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e5abc0 msgr2=0x55a860e5d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.812+0000 7f23b56a7640 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e5abc0 0x55a860e5d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23b56a7640 1 -- 172.21.15.77:0/601852833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55a50 msgr2=0x55a860e5a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23b56a7640 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55a50 0x55a860e5a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23b56a7640 1 -- 172.21.15.77:0/601852833 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a860e14500 con 0x55a860e55060 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23adffb640 1 -- 172.21.15.77:0/333910922 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f239c0f0650 con 0x7f23b0102a00 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55a50 0x55a860e5a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23b56a7640 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e55480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f23a8009fd0 tx=0x7f23a80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1345187ced81608d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23b56a7640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f23a00a2880 0x7f23a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f23affff640 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e5abc0 0x55a860e5d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f239affd640 1 -- 172.21.15.77:0/601852833 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a8004250 con 0x55a860e55060 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f239affd640 1 -- 172.21.15.77:0/601852833 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23a8003dd0 con 0x55a860e55060 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.813+0000 7f239affd640 1 -- 172.21.15.77:0/601852833 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a801bb50 con 0x55a860e55060 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 -- 172.21.15.77:0/601852833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 msgr2=0x55a860e55480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e55480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f23a8009fd0 tx=0x7f23a80118d0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b56a7640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f23a00a2880 0x7f23a00a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f23a8020000 tx=0x7f23a8021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 -- 172.21.15.77:0/601852833 shutdown_connections 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e5abc0 0x55a860e5d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55a50 0x55a860e5a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/601852833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e55480 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 -- 172.21.15.77:0/601852833 >> 172.21.15.77:0/601852833 conn(0x55a860e505d0 msgr2=0x55a860e509d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 -- 172.21.15.77:0/601852833 shutdown_connections 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.814+0000 7f23b9dcd740 1 -- 172.21.15.77:0/601852833 wait complete. 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 -- start start 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 Processor -- start 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55060 0x55a860e5f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55a50 0x55a860e5faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e66b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a860e5db70 con 0x55a860e5abc0 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a860d6d230 con 0x55a860e55060 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a860e5dcf0 con 0x55a860e55a50 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b9dcd740 10 client.0 _ll_register_callbacks cb 0x55a860da9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b4ea6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55060 0x55a860e5f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b4ea6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55060 0x55a860e5f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50224/0 (socket says 172.21.15.77:50224) 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b4ea6640 1 -- 172.21.15.77:0/2081972972 learned_addr learned my addr 172.21.15.77:0/2081972972 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23b56a7640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e66b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.815+0000 7f23affff640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55a50 0x55a860e5faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b4ea6640 1 -- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55a50 msgr2=0x55a860e5faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55a50 0x55a860e5faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b4ea6640 1 -- 172.21.15.77:0/2081972972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 msgr2=0x55a860e66b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e66b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b4ea6640 1 -- 172.21.15.77:0/2081972972 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a860d7c100 con 0x55a860e55060 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23affff640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55a50 0x55a860e5faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55060 0x55a860e5f5b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23a4012a10 tx=0x7f23a4012e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3425ab2c59a95cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f238ffff640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f238ffff640 1 -- 172.21.15.77:0/2081972972 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a400fa60 con 0x55a860e55060 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23a8003dd0 con 0x55a860e55060 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f238ffff640 1 -- 172.21.15.77:0/2081972972 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f23a400fc00 con 0x55a860e55060 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f238ffff640 1 -- 172.21.15.77:0/2081972972 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f23a400fe00 con 0x55a860e55060 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b9dcd740 10 client.25514 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.816+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a860ac53f0 con 0x55a860e55060 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f238ffff640 1 -- 172.21.15.77:0/2081972972 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23a8003dd0 con 0x55a860e55060 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f238ffff640 1 -- 172.21.15.77:0/2081972972 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f23a4010430 con 0x55a860e55060 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23b9dcd740 10 client.25514 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23b9dcd740 10 client.25514 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23b9dcd740 20 client.25514 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860e6b350 0x55a860e6d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23b9dcd740 4 client.25514 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a860dbe9d0 con 0x55a860e6b350 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23affff640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860e6b350 0x55a860e6d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f238ffff640 10 client.25514 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.817+0000 7f23affff640 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860e6b350 0x55a860e6d7f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d2fd1aa081d4aca server_cookie=1cc1483037d00eab in_seq=0 out_seq=0 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f238ffff640 1 -- 172.21.15.77:0/2081972972 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a860dbe9d0 con 0x55a860e6b350 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f238ffff640 10 client.25514 handle_command_reply: tid=0 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 client.25514 shutdown 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55060 msgr2=0x55a860e5f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55060 0x55a860e5f5b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23a4012a10 tx=0x7f23a4012e10 comp rx=0 tx=0).stop 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 shutdown_connections 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e66b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55a50 0x55a860e5faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860e6b350 0x55a860e6d7f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/2081972972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55060 0x55a860e5f5b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f238ffff640 0 client.25514 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 >> 172.21.15.77:0/2081972972 conn(0x55a860e505d0 msgr2=0x55a860e46fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 shutdown_connections 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 1 -- 172.21.15.77:0/2081972972 wait complete. 2024-03-16T09:42:58.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.818+0000 7f23b9dcd740 20 client.25514 trim_cache size 0 max 16384 2024-03-16T09:42:58.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 Processor -- start 2024-03-16T09:42:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 -- start start 2024-03-16T09:42:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55060 0x55a860e46e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e66af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e5abc0 0x55a860e720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f23b01a9300 con 0x55a860e55060 2024-03-16T09:42:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f23b01a6bd0 con 0x55a860e55a50 2024-03-16T09:42:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f23b01a4a20 con 0x55a860e5abc0 2024-03-16T09:42:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23affff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e66af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b4ea6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55060 0x55a860e46e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23affff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e66af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50228/0 (socket says 172.21.15.77:50228) 2024-03-16T09:42:58.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23affff640 1 -- 172.21.15.77:0/4290140513 learned_addr learned my addr 172.21.15.77:0/4290140513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b4ea6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55060 0x55a860e46e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43204/0 (socket says 172.21.15.77:43204) 2024-03-16T09:42:58.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.832+0000 7f23b56a7640 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e5abc0 0x55a860e720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f23affff640 1 -- 172.21.15.77:0/4290140513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e5abc0 msgr2=0x55a860e720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f23affff640 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e5abc0 0x55a860e720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f23affff640 1 -- 172.21.15.77:0/4290140513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55060 msgr2=0x55a860e46e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f23affff640 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55060 0x55a860e46e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f23affff640 1 -- 172.21.15.77:0/4290140513 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a860b8f7d0 con 0x55a860e55a50 2024-03-16T09:42:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55060 0x55a860e46e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f23affff640 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e66af0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f239c0bdbb0 tx=0x7f239c0035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dce4cedcf00b915 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f239a7fc640 1 -- 172.21.15.77:0/4290140513 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c0be5b0 con 0x55a860e55a50 2024-03-16T09:42:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f239a7fc640 1 -- 172.21.15.77:0/4290140513 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f239c0bdeb0 con 0x55a860e55a50 2024-03-16T09:42:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.833+0000 7f239a7fc640 1 -- 172.21.15.77:0/4290140513 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c005490 con 0x55a860e55a50 2024-03-16T09:42:58.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 -- 172.21.15.77:0/4290140513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 msgr2=0x55a860e66af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e66af0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f239c0bdbb0 tx=0x7f239c0035f0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 -- 172.21.15.77:0/4290140513 shutdown_connections 2024-03-16T09:42:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e5abc0 0x55a860e720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e66af0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/4290140513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e55060 0x55a860e46e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 -- 172.21.15.77:0/4290140513 >> 172.21.15.77:0/4290140513 conn(0x55a860e505d0 msgr2=0x55a860e46750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 -- 172.21.15.77:0/4290140513 shutdown_connections 2024-03-16T09:42:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 -- 172.21.15.77:0/4290140513 wait complete. 2024-03-16T09:42:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 -- start start 2024-03-16T09:42:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.834+0000 7f23b9dcd740 1 Processor -- start 2024-03-16T09:42:58.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e64370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e648b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b9dcd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e658c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f23b01a5090 con 0x55a860e5abc0 2024-03-16T09:42:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f23b01a6500 con 0x55a860e55a50 2024-03-16T09:42:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b9dcd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f23b01a5760 con 0x55a860e55060 2024-03-16T09:42:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b9dcd740 10 client.0 _ll_register_callbacks cb 0x7f23a4009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e648b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50238/0 (socket says 172.21.15.77:50238) 2024-03-16T09:42:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 -- 172.21.15.77:0/1496282121 learned_addr learned my addr 172.21.15.77:0/1496282121 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b56a7640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e64370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 -- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 msgr2=0x55a860e64370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e64370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 -- 172.21.15.77:0/1496282121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 msgr2=0x55a860e658c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e658c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23affff640 1 -- 172.21.15.77:0/1496282121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a860e65eb0 con 0x55a860e55a50 2024-03-16T09:42:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e64370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.835+0000 7f23b56a7640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e658c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23affff640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e648b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f239c0bde40 tx=0x7f239c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3251358199db72d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f238ffff640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:42:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f238ffff640 1 -- 172.21.15.77:0/1496282121 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c0c6650 con 0x55a860e55a50 2024-03-16T09:42:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f238ffff640 1 -- 172.21.15.77:0/1496282121 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f239c0c61c0 con 0x55a860e55a50 2024-03-16T09:42:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f239c0bdeb0 con 0x55a860e55a50 2024-03-16T09:42:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f238ffff640 1 -- 172.21.15.77:0/1496282121 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f239c005690 con 0x55a860e55a50 2024-03-16T09:42:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23b9dcd740 10 client.25520 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a860ed18e0 con 0x55a860e55a50 2024-03-16T09:42:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f238ffff640 1 -- 172.21.15.77:0/1496282121 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f239c0bdeb0 con 0x55a860e55a50 2024-03-16T09:42:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f238ffff640 1 -- 172.21.15.77:0/1496282121 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f23a4009ce0 con 0x55a860e55a50 2024-03-16T09:42:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23b9dcd740 10 client.25520 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:58.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23b9dcd740 10 client.25520 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:42:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23b9dcd740 20 client.25520 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.836+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860ed2250 0x55a860ed4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f23b9dcd740 4 client.25520 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:42:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a860db8d20 con 0x55a860ed2250 2024-03-16T09:42:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860ed2250 0x55a860ed4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f238ffff640 10 client.25520 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f23b4ea6640 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860ed2250 0x55a860ed4670 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58a0d24eea1c9958 server_cookie=d1856d73d796e3b in_seq=0 out_seq=0 2024-03-16T09:42:58.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f238ffff640 1 -- 172.21.15.77:0/1496282121 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a860db8d20 con 0x55a860ed2250 2024-03-16T09:42:58.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f238ffff640 10 client.25520 handle_command_reply: tid=0 2024-03-16T09:42:58.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f23b9dcd740 1 client.25520 shutdown 2024-03-16T09:42:58.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.837+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 msgr2=0x55a860e648b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e648b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f239c0bde40 tx=0x7f239c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 shutdown_connections 2024-03-16T09:42:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a860e5abc0 0x55a860e658c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55a860ed2250 0x55a860ed4670 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a860e55a50 0x55a860e648b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f238ffff640 0 client.25520 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:42:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 --2- 172.21.15.77:0/1496282121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a860e55060 0x55a860e64370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 >> 172.21.15.77:0/1496282121 conn(0x55a860e505d0 msgr2=0x55a860e46fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 shutdown_connections 2024-03-16T09:42:58.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 1 -- 172.21.15.77:0/1496282121 wait complete. 2024-03-16T09:42:58.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.838+0000 7f23b9dcd740 20 client.25520 trim_cache size 0 max 16384 2024-03-16T09:42:58.860 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:42:58.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f23a00a2880 msgr2=0x7f23a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f23a00a2880 0x7f23a00a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f23a8020000 tx=0x7f23a8021000 comp rx=0 tx=0).stop 2024-03-16T09:42:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0102a00 msgr2=0x7f23b0073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0102a00 0x7f23b0073bd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f239c006010 tx=0x7f239c005e40 comp rx=0 tx=0).stop 2024-03-16T09:42:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 shutdown_connections 2024-03-16T09:42:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f23a00a2880 0x7f23a00a4d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b01033c0 0x7f23b0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23b0102a00 0x7f23b0073bd0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 --2- 172.21.15.77:0/333910922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23b0101810 0x7f23b0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 >> 172.21.15.77:0/333910922 conn(0x7f23b00fcd80 msgr2=0x7f23b0105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 shutdown_connections 2024-03-16T09:42:58.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.839+0000 7f23b710a640 1 -- 172.21.15.77:0/333910922 wait complete. 2024-03-16T09:42:58.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 -- 172.21.15.77:0/3381829125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee8102a00 msgr2=0x7f0ee8102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 --2- 172.21.15.77:0/3381829125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee8102a00 0x7f0ee8102e80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009860 tx=0x7f0edc02bb70 comp rx=0 tx=0).stop 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 -- 172.21.15.77:0/3381829125 shutdown_connections 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 --2- 172.21.15.77:0/3381829125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ee81033c0 0x7f0ee8109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 --2- 172.21.15.77:0/3381829125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee8102a00 0x7f0ee8102e80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 --2- 172.21.15.77:0/3381829125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 0x7f0ee8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 -- 172.21.15.77:0/3381829125 >> 172.21.15.77:0/3381829125 conn(0x7f0ee80fcd80 msgr2=0x7f0ee80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 -- 172.21.15.77:0/3381829125 shutdown_connections 2024-03-16T09:42:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 -- 172.21.15.77:0/3381829125 wait complete. 2024-03-16T09:42:58.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.972+0000 7f0eededc640 1 Processor -- start 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0eededc640 1 -- start start 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0eededc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 0x7f0ee8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0eededc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ee8102a00 0x7f0ee8197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0eededc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee81033c0 0x7f0ee819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0eededc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee810d320 con 0x7f0ee8101810 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0eededc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee810d1a0 con 0x7f0ee8102a00 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0eededc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee810d4a0 con 0x7f0ee81033c0 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 0x7f0ee8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 0x7f0ee8197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43232/0 (socket says 172.21.15.77:43232) 2024-03-16T09:42:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 -- 172.21.15.77:0/316673488 learned_addr learned my addr 172.21.15.77:0/316673488 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ee8102a00 0x7f0ee8197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee7fff640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee81033c0 0x7f0ee819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 -- 172.21.15.77:0/316673488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee81033c0 msgr2=0x7f0ee819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee81033c0 0x7f0ee819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 -- 172.21.15.77:0/316673488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ee8102a00 msgr2=0x7f0ee8197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ee8102a00 0x7f0ee8197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 -- 172.21.15.77:0/316673488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee8073a90 con 0x7f0ee8101810 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ee8102a00 0x7f0ee8197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.973+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 0x7f0ee8197a30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800e9f0 tx=0x7f0ed800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.974+0000 7f0ee4ff9640 1 -- 172.21.15.77:0/316673488 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800ccc0 con 0x7f0ee8101810 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.974+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee8073d80 con 0x7f0ee8101810 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.974+0000 7f0ee4ff9640 1 -- 172.21.15.77:0/316673488 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ed80035f0 con 0x7f0ee8101810 2024-03-16T09:42:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.974+0000 7f0ee4ff9640 1 -- 172.21.15.77:0/316673488 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800ce60 con 0x7f0ee8101810 2024-03-16T09:42:58.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.974+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee80742f0 con 0x7f0ee8101810 2024-03-16T09:42:58.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.976+0000 7f0ee4ff9640 1 -- 172.21.15.77:0/316673488 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ed80037a0 con 0x7f0ee8101810 2024-03-16T09:42:58.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.976+0000 7f0ef0ba1740 1 Processor -- start 2024-03-16T09:42:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ee4ff9640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ed00a2900 0x7f0ed00a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ee4ff9640 1 -- 172.21.15.77:0/316673488 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0ed80cec50 con 0x7f0ee8101810 2024-03-16T09:42:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ef0ba1740 1 -- start start 2024-03-16T09:42:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ee7fff640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ed00a2900 0x7f0ed00a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafc240 0x55988fb00d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb03840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55988fa60490 con 0x55988fafc240 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55988fa08c90 con 0x55988fb013b0 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55988fa5c530 con 0x55988fafb850 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ee6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafc240 0x55988fb00d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ee6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafc240 0x55988fb00d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43242/0 (socket says 172.21.15.77:43242) 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/139600938 learned_addr learned my addr 172.21.15.77:0/139600938 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.977+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/139600938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 msgr2=0x55988fafbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/139600938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 msgr2=0x55988fb03840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb03840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/139600938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55988fac3500 con 0x55988fafc240 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee7fff640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ed00a2900 0x7f0ed00a4dc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0005fd0 tx=0x7f0ee0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafc240 0x55988fb00d40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009860 tx=0x7f0edc005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9182ebc68e811bae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ecdffb640 1 -- 172.21.15.77:0/139600938 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc039040 con 0x55988fafc240 2024-03-16T09:42:58.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ecdffb640 1 -- 172.21.15.77:0/139600938 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0edc034dc0 con 0x55988fafc240 2024-03-16T09:42:58.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.978+0000 7f0ecdffb640 1 -- 172.21.15.77:0/139600938 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc0035f0 con 0x55988fafc240 2024-03-16T09:42:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/139600938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafc240 msgr2=0x55988fb00d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafc240 0x55988fb00d40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009860 tx=0x7f0edc005db0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/139600938 shutdown_connections 2024-03-16T09:42:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb03840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafc240 0x55988fb00d40 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/139600938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/139600938 >> 172.21.15.77:0/139600938 conn(0x55988faf6dc0 msgr2=0x55988faf71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/139600938 shutdown_connections 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.979+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/139600938 wait complete. 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 -- start start 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 Processor -- start 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb05ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb06420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb0d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55988fb04300 con 0x55988fafb850 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55988fa1c230 con 0x55988fb013b0 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55988fb04480 con 0x55988fb0b450 2024-03-16T09:42:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ef0ba1740 10 client.0 _ll_register_callbacks cb 0x55988fa58010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb05ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb05ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43250/0 (socket says 172.21.15.77:43250) 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ee77fe640 1 -- 172.21.15.77:0/4157036480 learned_addr learned my addr 172.21.15.77:0/4157036480 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ee7fff640 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb0d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb06420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ee77fe640 1 -- 172.21.15.77:0/4157036480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 msgr2=0x55988fb0d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.980+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb0d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ee77fe640 1 -- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 msgr2=0x55988fb06420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb06420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ee77fe640 1 -- 172.21.15.77:0/4157036480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55988fa2b100 con 0x55988fafb850 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb05ee0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009530 tx=0x7f0ed8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef597e15fba2358 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ec6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/4157036480 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80b4020 con 0x55988fafb850 2024-03-16T09:42:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/4157036480 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0ed80a3ab0 con 0x55988fafb850 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0edc034dc0 con 0x55988fafb850 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/4157036480 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0ed809f6f0 con 0x55988fafb850 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ef0ba1740 10 client.15645 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55988f7743f0 con 0x55988fafb850 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/4157036480 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0edc034dc0 con 0x55988fafb850 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.981+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/4157036480 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0ed809a9f0 con 0x55988fafb850 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ef0ba1740 10 client.15645 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ef0ba1740 10 client.15645 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ef0ba1740 20 client.15645 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb144d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ef0ba1740 4 client.15645 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55988fa6d9d0 con 0x55988fb12080 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ec6ffd640 10 client.15645 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.982+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb144d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb7a3f93dfe27dfd server_cookie=c09757261d5af906 in_seq=0 out_seq=0 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/4157036480 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55988fa6d9d0 con 0x55988fb12080 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ec6ffd640 10 client.15645 handle_command_reply: tid=0 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 client.15645 shutdown 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 msgr2=0x55988fb05ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb05ee0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009530 tx=0x7f0ed8003b90 comp rx=0 tx=0).stop 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 shutdown_connections 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb144d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb0d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ec6ffd640 0 client.15645 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb06420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/4157036480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb05ee0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 >> 172.21.15.77:0/4157036480 conn(0x55988faf6dc0 msgr2=0x55988faf5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 shutdown_connections 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.983+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/4157036480 wait complete. 2024-03-16T09:42:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:58.984+0000 7f0ef0ba1740 20 client.15645 trim_cache size 0 max 16384 2024-03-16T09:42:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:58 smithi077 ceph-mon[32171]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:59.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:42:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2881065172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:42:59.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.009+0000 7f0ef0ba1740 1 Processor -- start 2024-03-16T09:42:59.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.009+0000 7f0ef0ba1740 1 -- start start 2024-03-16T09:42:59.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.009+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.009+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb05af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fb0b450 0x55988fb06030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee81ad640 con 0x55988fb0b450 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee81aaf10 con 0x55988fb013b0 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ef0ba1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee81a8d60 con 0x55988fafb850 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb05af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb05af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50290/0 (socket says 172.21.15.77:50290) 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/1770569163 learned_addr learned my addr 172.21.15.77:0/1770569163 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37380/0 (socket says 172.21.15.77:37380) 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee7fff640 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fb0b450 0x55988fb06030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/1770569163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 msgr2=0x55988fafcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/1770569163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fb0b450 msgr2=0x55988fb06030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fb0b450 0x55988fb06030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 -- 172.21.15.77:0/1770569163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55988f83e7d0 con 0x55988fb013b0 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee7fff640 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fb0b450 0x55988fb06030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.010+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb05af0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0edc02c0b0 tx=0x7f0edc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8386b61d74240f7c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.011+0000 7f0ecd7fa640 1 -- 172.21.15.77:0/1770569163 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc039040 con 0x55988fb013b0 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.011+0000 7f0ecd7fa640 1 -- 172.21.15.77:0/1770569163 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0edc0513c0 con 0x55988fb013b0 2024-03-16T09:42:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.011+0000 7f0ecd7fa640 1 -- 172.21.15.77:0/1770569163 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc02c890 con 0x55988fb013b0 2024-03-16T09:42:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/1770569163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 msgr2=0x55988fb05af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb05af0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0edc02c0b0 tx=0x7f0edc000bc0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/1770569163 shutdown_connections 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fb0b450 0x55988fb06030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb05af0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/1770569163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fafb850 0x55988fafcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/1770569163 >> 172.21.15.77:0/1770569163 conn(0x55988faf6dc0 msgr2=0x55988fafd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/1770569163 shutdown_connections 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/1770569163 wait complete. 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.012+0000 7f0ef0ba1740 1 -- start start 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ef0ba1740 1 Processor -- start 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb23000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb23540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ef0ba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb24500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb23000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb23000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43268/0 (socket says 172.21.15.77:43268) 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ee6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb23540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ee77fe640 1 -- 172.21.15.77:0/689668858 learned_addr learned my addr 172.21.15.77:0/689668858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee81a93d0 con 0x55988fafb850 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee81aa840 con 0x55988fb013b0 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee81a9aa0 con 0x55988fb0b450 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.013+0000 7f0ee7fff640 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb24500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ee77fe640 1 -- 172.21.15.77:0/689668858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 msgr2=0x55988fb24500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb24500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ee77fe640 1 -- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 msgr2=0x55988fb23540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb23540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ef0ba1740 10 client.0 _ll_register_callbacks cb 0x7f0ed80971d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ee77fe640 1 -- 172.21.15.77:0/689668858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55988fb24af0 con 0x55988fafb850 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb23540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0edc0513c0 con 0x55988fafb850 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ee77fe640 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb23000 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0ed809ade0 tx=0x7f0ed80b4410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa45f72d05dda69 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ec6ffd640 10 client.15651 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/689668858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80b4040 con 0x55988fafb850 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/689668858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0ed80a3ac0 con 0x55988fafb850 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/689668858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0ed809af00 con 0x55988fafb850 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ef0ba1740 10 client.15651 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.014+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55988fb77f80 con 0x55988fafb850 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/689668858 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0ed8083050 con 0x55988fafb850 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ef0ba1740 10 client.15651 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ef0ba1740 10 client.15651 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ef0ba1740 20 client.15651 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb7a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ef0ba1740 4 client.15651 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55988fa67d20 con 0x55988fb12080 2024-03-16T09:42:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.015+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb7a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ec6ffd640 10 client.15651 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ee6ffd640 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb7a800 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56deb960953f5719 server_cookie=fe0035298747712c in_seq=0 out_seq=0 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ec6ffd640 1 -- 172.21.15.77:0/689668858 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55988fa67d20 con 0x55988fb12080 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ec6ffd640 10 client.15651 handle_command_reply: tid=0 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 client.15651 shutdown 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 msgr2=0x55988fb23000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb23000 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0ed809ade0 tx=0x7f0ed80b4410 comp rx=0 tx=0).stop 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 shutdown_connections 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x55988fb12080 0x55988fb7a800 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55988fb0b450 0x55988fb24500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ec6ffd640 0 client.15651 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55988fb013b0 0x55988fb23540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 --2- 172.21.15.77:0/689668858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55988fafb850 0x55988fb23000 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 >> 172.21.15.77:0/689668858 conn(0x55988faf6dc0 msgr2=0x55988fb05af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 shutdown_connections 2024-03-16T09:42:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.016+0000 7f0ef0ba1740 1 -- 172.21.15.77:0/689668858 wait complete. 2024-03-16T09:42:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.017+0000 7f0ef0ba1740 20 client.15651 trim_cache size 0 max 16384 2024-03-16T09:42:59.025 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:42:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.017+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ed00a2900 msgr2=0x7f0ed00a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.017+0000 7f0eededc640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ed00a2900 0x7f0ed00a4dc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0005fd0 tx=0x7f0ee0009500 comp rx=0 tx=0).stop 2024-03-16T09:42:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.017+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 msgr2=0x7f0ee8197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.017+0000 7f0eededc640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 0x7f0ee8197a30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800e9f0 tx=0x7f0ed800eeb0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.017+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 shutdown_connections 2024-03-16T09:42:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.017+0000 7f0eededc640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f0ed00a2900 0x7f0ed00a4dc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.018+0000 7f0eededc640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ee81033c0 0x7f0ee819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.018+0000 7f0eededc640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ee8102a00 0x7f0ee8197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.018+0000 7f0eededc640 1 --2- 172.21.15.77:0/316673488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ee8101810 0x7f0ee8197a30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.018+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 >> 172.21.15.77:0/316673488 conn(0x7f0ee80fcd80 msgr2=0x7f0ee8105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.018+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 shutdown_connections 2024-03-16T09:42:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.018+0000 7f0eededc640 1 -- 172.21.15.77:0/316673488 wait complete. 2024-03-16T09:42:59.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T09:42:59.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 -- 172.21.15.77:0/3011488914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10102a00 msgr2=0x7f6d10102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 --2- 172.21.15.77:0/3011488914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10102a00 0x7f6d10102e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009900 tx=0x7f6d0402bc30 comp rx=0 tx=0).stop 2024-03-16T09:42:59.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 -- 172.21.15.77:0/3011488914 shutdown_connections 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 --2- 172.21.15.77:0/3011488914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d101033c0 0x7f6d10109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 --2- 172.21.15.77:0/3011488914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10102a00 0x7f6d10102e80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 --2- 172.21.15.77:0/3011488914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10101810 0x7f6d10101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 -- 172.21.15.77:0/3011488914 >> 172.21.15.77:0/3011488914 conn(0x7f6d100fcd80 msgr2=0x7f6d100ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 -- 172.21.15.77:0/3011488914 shutdown_connections 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.137+0000 7f6d16220640 1 -- 172.21.15.77:0/3011488914 wait complete. 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 Processor -- start 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 -- start start 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10101810 0x7f6d10195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10102a00 0x7f6d10195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d101033c0 0x7f6d1019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010b1b0 con 0x7f6d101033c0 2024-03-16T09:42:59.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010b030 con 0x7f6d10101810 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d16220640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010b330 con 0x7f6d10102a00 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10101810 0x7f6d10195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10102a00 0x7f6d10195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d101033c0 0x7f6d1019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10102a00 0x7f6d10195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37404/0 (socket says 172.21.15.77:37404) 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 -- 172.21.15.77:0/2324601322 learned_addr learned my addr 172.21.15.77:0/2324601322 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.138+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10101810 0x7f6d10195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50322/0 (socket says 172.21.15.77:50322) 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 -- 172.21.15.77:0/2324601322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10101810 msgr2=0x7f6d10195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10101810 0x7f6d10195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 -- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d101033c0 msgr2=0x7f6d1019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d101033c0 0x7f6d1019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 -- 172.21.15.77:0/2324601322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1019d430 con 0x7f6d10102a00 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d101033c0 0x7f6d1019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0effd640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10102a00 0x7f6d10195d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6d04000c00 tx=0x7f6d040040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0cff9640 1 -- 172.21.15.77:0/2324601322 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04043020 con 0x7f6d10102a00 2024-03-16T09:42:59.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1019d6c0 con 0x7f6d10102a00 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0cff9640 1 -- 172.21.15.77:0/2324601322 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d040044c0 con 0x7f6d10102a00 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d0cff9640 1 -- 172.21.15.77:0/2324601322 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04003c60 con 0x7f6d10102a00 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.139+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d1019dc60 con 0x7f6d10102a00 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.142+0000 7f6d0cff9640 1 -- 172.21.15.77:0/2324601322 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6d04046020 con 0x7f6d10102a00 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.142+0000 7f6d18e0d740 1 Processor -- start 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d18e0d740 1 -- start start 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f7850 0x55b38e2f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f8240 0x55b38e2fcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e2ff840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b38e25c490 con 0x55b38e2f8240 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b38e204c90 con 0x55b38e2fd3b0 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b38e258530 con 0x55b38e2f7850 2024-03-16T09:42:59.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f7850 0x55b38e2f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f8240 0x55b38e2fcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f7850 0x55b38e2f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37420/0 (socket says 172.21.15.77:37420) 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0ffff640 1 -- 172.21.15.77:0/1222500146 learned_addr learned my addr 172.21.15.77:0/1222500146 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0cff9640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6cf80a2800 0x7f6cf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0f7fe640 1 -- 172.21.15.77:0/1222500146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f7850 msgr2=0x55b38e2f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0f7fe640 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f7850 0x55b38e2f7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0f7fe640 1 -- 172.21.15.77:0/1222500146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 msgr2=0x55b38e2ff840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0f7fe640 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e2ff840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0f7fe640 1 -- 172.21.15.77:0/1222500146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b38e2bf500 con 0x55b38e2f8240 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f7850 0x55b38e2f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6cf80a2800 0x7f6cf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.143+0000 7f6d0cff9640 1 -- 172.21.15.77:0/2324601322 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6d040f0c50 con 0x7f6d10102a00 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.144+0000 7f6d0f7fe640 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f8240 0x55b38e2fcd40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009bd0 tx=0x7f6d00011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed083df5bc5549e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.144+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/1222500146 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00004230 con 0x55b38e2f8240 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.144+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/1222500146 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d00009e40 con 0x55b38e2f8240 2024-03-16T09:42:59.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.144+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6cf80a2800 0x7f6cf80a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005c40 tx=0x7f6d08004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.144+0000 7f6cf5ffb640 1 -- 172.21.15.77:0/1222500146 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00019790 con 0x55b38e2f8240 2024-03-16T09:42:59.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1222500146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f8240 msgr2=0x55b38e2fcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f8240 0x55b38e2fcd40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009bd0 tx=0x7f6d00011810 comp rx=0 tx=0).stop 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1222500146 shutdown_connections 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e2ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f8240 0x55b38e2fcd40 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1222500146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f7850 0x55b38e2f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1222500146 >> 172.21.15.77:0/1222500146 conn(0x55b38e2f2dc0 msgr2=0x55b38e2f31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1222500146 shutdown_connections 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1222500146 wait complete. 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 -- start start 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.145+0000 7f6d18e0d740 1 Processor -- start 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e301df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e302330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b38e300360 con 0x55b38e2f7850 2024-03-16T09:42:59.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b38e218230 con 0x55b38e2fd3b0 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b38e3004e0 con 0x55b38e2f8240 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d18e0d740 10 client.0 _ll_register_callbacks cb 0x55b38e254010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e301df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e301df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43296/0 (socket says 172.21.15.77:43296) 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0f7fe640 1 -- 172.21.15.77:0/2412334208 learned_addr learned my addr 172.21.15.77:0/2412334208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0effd640 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e302330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0ffff640 1 -- 172.21.15.77:0/2412334208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 msgr2=0x55b38e302330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e302330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0ffff640 1 -- 172.21.15.77:0/2412334208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 msgr2=0x55b38e301df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e301df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0ffff640 1 -- 172.21.15.77:0/2412334208 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b38e227100 con 0x55b38e2fd3b0 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.146+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005fd0 tx=0x7f6d08011040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1375da88eee59c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6ceeffd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6ceeffd640 1 -- 172.21.15.77:0/2412334208 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08026640 con 0x55b38e2fd3b0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6ceeffd640 1 -- 172.21.15.77:0/2412334208 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6d080267e0 con 0x55b38e2fd3b0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b38e2699d0 con 0x55b38e2fd3b0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6ceeffd640 1 -- 172.21.15.77:0/2412334208 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6d0801be10 con 0x55b38e2fd3b0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6d18e0d740 10 client.25544 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b38df703f0 con 0x55b38e2fd3b0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.147+0000 7f6ceeffd640 1 -- 172.21.15.77:0/2412334208 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b38e2699d0 con 0x55b38e2fd3b0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6ceeffd640 1 -- 172.21.15.77:0/2412334208 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6d08032b90 con 0x55b38e2fd3b0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d18e0d740 10 client.25544 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d18e0d740 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d18e0d740 20 client.25544 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e30df50 0x55b38e3103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d18e0d740 4 client.25544 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b38e263d20 con 0x55b38e30df50 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d0f7fe640 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e30df50 0x55b38e3103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6ceeffd640 10 client.25544 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.148+0000 7f6d0f7fe640 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e30df50 0x55b38e3103a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e614f0e53970f92 server_cookie=580229c70c024965 in_seq=0 out_seq=0 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6ceeffd640 1 -- 172.21.15.77:0/2412334208 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b38e263d20 con 0x55b38e30df50 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6ceeffd640 10 client.25544 handle_command_reply: tid=0 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 client.25544 shutdown 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 msgr2=0x55b38e309370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005fd0 tx=0x7f6d08011040 comp rx=0 tx=0).stop 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 shutdown_connections 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e30df50 0x55b38e3103a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309370 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e302330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/2412334208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e301df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 >> 172.21.15.77:0/2412334208 conn(0x55b38e2f2dc0 msgr2=0x55b38e2f1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6ceeffd640 0 client.25544 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 shutdown_connections 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.149+0000 7f6d18e0d740 1 -- 172.21.15.77:0/2412334208 wait complete. 2024-03-16T09:42:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.150+0000 7f6d18e0d740 20 client.25544 trim_cache size 0 max 16384 2024-03-16T09:42:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 Processor -- start 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 -- start start 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e2f97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e2f9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d101a9170 con 0x55b38e2f7850 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6d101a6a40 con 0x55b38e2fd3b0 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d18e0d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d101a4890 con 0x55b38e2f8240 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e2f9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e2f97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e2f9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37436/0 (socket says 172.21.15.77:37436) 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 -- 172.21.15.77:0/405236715 learned_addr learned my addr 172.21.15.77:0/405236715 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 -- 172.21.15.77:0/405236715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 msgr2=0x55b38e309290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 -- 172.21.15.77:0/405236715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 msgr2=0x55b38e2f97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e2f97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.163+0000 7f6d0effd640 1 -- 172.21.15.77:0/405236715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b38e03a7d0 con 0x55b38e2f8240 2024-03-16T09:42:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.164+0000 7f6d0effd640 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e2f9d10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d040f0fb0 tx=0x7f6d040a8040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1c180ae66cb4177 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.164+0000 7f6cf57fa640 1 -- 172.21.15.77:0/405236715 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d040b9040 con 0x55b38e2f8240 2024-03-16T09:42:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.164+0000 7f6cf57fa640 1 -- 172.21.15.77:0/405236715 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d040035f0 con 0x55b38e2f8240 2024-03-16T09:42:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.164+0000 7f6cf57fa640 1 -- 172.21.15.77:0/405236715 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d040bb6a0 con 0x55b38e2f8240 2024-03-16T09:42:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 -- 172.21.15.77:0/405236715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 msgr2=0x55b38e2f9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e2f9d10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d040f0fb0 tx=0x7f6d040a8040 comp rx=0 tx=0).stop 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 -- 172.21.15.77:0/405236715 shutdown_connections 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2fd3b0 0x55b38e309290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e2f9d10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/405236715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2f7850 0x55b38e2f97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 -- 172.21.15.77:0/405236715 >> 172.21.15.77:0/405236715 conn(0x55b38e2f2dc0 msgr2=0x55b38e301df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 -- 172.21.15.77:0/405236715 shutdown_connections 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 -- 172.21.15.77:0/405236715 wait complete. 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 -- start start 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.165+0000 7f6d18e0d740 1 Processor -- start 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2f7850 0x55b38e308da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e3092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d18e0d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2fd3b0 0x55b38e305980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e3092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d0f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2f7850 0x55b38e308da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e3092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37438/0 (socket says 172.21.15.77:37438) 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d0effd640 1 -- 172.21.15.77:0/1723106423 learned_addr learned my addr 172.21.15.77:0/1723106423 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d101a4f00 con 0x55b38e2fd3b0 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6d101a6370 con 0x55b38e2f7850 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d101a55d0 con 0x55b38e2f8240 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d0ffff640 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2fd3b0 0x55b38e305980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.166+0000 7f6d18e0d740 10 client.0 _ll_register_callbacks cb 0x7f6d08009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d0effd640 1 -- 172.21.15.77:0/1723106423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2f7850 msgr2=0x55b38e308da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d0effd640 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2f7850 0x55b38e308da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d0effd640 1 -- 172.21.15.77:0/1723106423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2fd3b0 msgr2=0x55b38e305980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d0effd640 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2fd3b0 0x55b38e305980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d0effd640 1 -- 172.21.15.77:0/1723106423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b38e3061b0 con 0x55b38e2f8240 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d0effd640 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e3092e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d040ee760 tx=0x7f6d0402c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17df657fb7843124 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6ceeffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-03-16T09:42:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6ceeffd640 1 -- 172.21.15.77:0/1723106423 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d040da070 con 0x55b38e2f8240 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6ceeffd640 1 -- 172.21.15.77:0/1723106423 <== mon.1 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6d04003e00 con 0x55b38e2f8240 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b38e1e16a0 con 0x55b38e2f8240 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6ceeffd640 1 -- 172.21.15.77:0/1723106423 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6d040b9040 con 0x55b38e2f8240 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d18e0d740 10 client.25477 fetch_fsmap learned FSMap version 21 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b38e373120 con 0x55b38e2f8240 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.167+0000 7f6ceeffd640 1 -- 172.21.15.77:0/1723106423 <== mon.1 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b38e1e16a0 con 0x55b38e2f8240 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6ceeffd640 1 -- 172.21.15.77:0/1723106423 <== mon.1 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6d040a6b90 con 0x55b38e2f8240 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d18e0d740 10 client.25477 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d18e0d740 10 client.25477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d18e0d740 20 client.25477 populate_metadata read hostname 'smithi077' 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e3738a0 0x55b38e375cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d18e0d740 4 client.25477 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:42:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b38e22ddc0 con 0x55b38e3738a0 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d0f7fe640 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e3738a0 0x55b38e375cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6ceeffd640 10 client.25477 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.168+0000 7f6d0f7fe640 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e3738a0 0x55b38e375cc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=565490f03a9e562c server_cookie=8f1760a35337f815 in_seq=0 out_seq=0 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6ceeffd640 1 -- 172.21.15.77:0/1723106423 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b38e22ddc0 con 0x55b38e3738a0 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6ceeffd640 10 client.25477 handle_command_reply: tid=0 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 client.25477 shutdown 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 msgr2=0x55b38e3092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e3092e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d040ee760 tx=0x7f6d0402c470 comp rx=0 tx=0).stop 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 shutdown_connections 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b38e3738a0 0x55b38e375cc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b38e2fd3b0 0x55b38e305980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6ceeffd640 0 client.25477 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b38e2f8240 0x55b38e3092e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 --2- 172.21.15.77:0/1723106423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b38e2f7850 0x55b38e308da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 >> 172.21.15.77:0/1723106423 conn(0x55b38e2f2dc0 msgr2=0x55b38e301db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 shutdown_connections 2024-03-16T09:42:59.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 1 -- 172.21.15.77:0/1723106423 wait complete. 2024-03-16T09:42:59.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.169+0000 7f6d18e0d740 20 client.25477 trim_cache size 0 max 16384 2024-03-16T09:42:59.177 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6cf80a2800 msgr2=0x7f6cf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6cf80a2800 0x7f6cf80a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005c40 tx=0x7f6d08004a40 comp rx=0 tx=0).stop 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10102a00 msgr2=0x7f6d10195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10102a00 0x7f6d10195d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6d04000c00 tx=0x7f6d040040c0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 shutdown_connections 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6cf80a2800 0x7f6cf80a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d101033c0 0x7f6d1019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d10102a00 0x7f6d10195d50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 --2- 172.21.15.77:0/2324601322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d10101810 0x7f6d10195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 >> 172.21.15.77:0/2324601322 conn(0x7f6d100fcd80 msgr2=0x7f6d10105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:42:59.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 shutdown_connections 2024-03-16T09:42:59.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:42:59.170+0000 7f6d16220640 1 -- 172.21.15.77:0/2324601322 wait complete. 2024-03-16T09:42:59.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:58 smithi119 ceph-mon[35632]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:59.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:42:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2881065172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:42:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:58 smithi129 ceph-mon[35841]: pgmap v378: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:42:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:42:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2881065172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:43:00.187 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T09:43:00.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:43:00.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T09:43:00.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 -- 172.21.15.77:0/2365059203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d8102290 msgr2=0x7f64d8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 --2- 172.21.15.77:0/2365059203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d8102290 0x7f64d8108b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009be0 tx=0x7f64d002bef0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 -- 172.21.15.77:0/2365059203 shutdown_connections 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 --2- 172.21.15.77:0/2365059203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d8102290 0x7f64d8108b20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 --2- 172.21.15.77:0/2365059203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64d81018d0 0x7f64d8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 --2- 172.21.15.77:0/2365059203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64d81006e0 0x7f64d8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 -- 172.21.15.77:0/2365059203 >> 172.21.15.77:0/2365059203 conn(0x7f64d80fbc70 msgr2=0x7f64d80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 -- 172.21.15.77:0/2365059203 shutdown_connections 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 -- 172.21.15.77:0/2365059203 wait complete. 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.332+0000 7f64de47e640 1 Processor -- start 2024-03-16T09:43:00.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64de47e640 1 -- start start 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64de47e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d81006e0 0x7f64d810b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64de47e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64d81018d0 0x7f64d81096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64de47e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64d8102290 0x7f64d8109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64de47e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f64d810d330 con 0x7f64d8102290 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64de47e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f64d810d1b0 con 0x7f64d81006e0 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64de47e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64d810d4b0 con 0x7f64d81018d0 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d81006e0 0x7f64d810b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d81006e0 0x7f64d810b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50378/0 (socket says 172.21.15.77:50378) 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 -- 172.21.15.77:0/3300876233 learned_addr learned my addr 172.21.15.77:0/3300876233 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64dca1b640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64d8102290 0x7f64d8109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d77fe640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64d81018d0 0x7f64d81096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 -- 172.21.15.77:0/3300876233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64d81018d0 msgr2=0x7f64d81096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64d81018d0 0x7f64d81096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 -- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64d8102290 msgr2=0x7f64d8109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64d8102290 0x7f64d8109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.333+0000 7f64d7fff640 1 -- 172.21.15.77:0/3300876233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d810a470 con 0x7f64d81006e0 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.334+0000 7f64d7fff640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d81006e0 0x7f64d810b060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64c800e9a0 tx=0x7f64c800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.334+0000 7f64d57fa640 1 -- 172.21.15.77:0/3300876233 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c800cd80 con 0x7f64d81006e0 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.334+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d81a1b30 con 0x7f64d81006e0 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.334+0000 7f64d57fa640 1 -- 172.21.15.77:0/3300876233 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64c80108e0 con 0x7f64d81006e0 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.334+0000 7f64d57fa640 1 -- 172.21.15.77:0/3300876233 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8018950 con 0x7f64d81006e0 2024-03-16T09:43:00.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.334+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d81a20d0 con 0x7f64d81006e0 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.336+0000 7f64d57fa640 1 -- 172.21.15.77:0/3300876233 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f64c8010a90 con 0x7f64d81006e0 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.336+0000 7f64e1069740 1 Processor -- start 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64e1069740 1 -- start start 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64e1069740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b485ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64d57fa640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f64c00a2880 0x7f64c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64e1069740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b4862e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64e1069740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b48634f0 0x5584b4865980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64e1069740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5584b47c20d0 con 0x5584b485e380 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64e1069740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5584b476ac70 con 0x5584b485d990 2024-03-16T09:43:00.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64e1069740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5584b47be170 con 0x5584b48634f0 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64dca1b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b485ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64dca1b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b485ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50390/0 (socket says 172.21.15.77:50390) 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64d57fa640 1 -- 172.21.15.77:0/3300876233 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f64c80ced60 con 0x7f64d81006e0 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64dca1b640 1 -- 172.21.15.77:0/590591933 learned_addr learned my addr 172.21.15.77:0/590591933 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64d7fff640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f64c00a2880 0x7f64c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64d77fe640 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b4862e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64dca1b640 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b48634f0 0x5584b4865980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.337+0000 7f64d7fff640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f64c00a2880 0x7f64c00a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f64c809b410 tx=0x7f64c8018420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64dca1b640 1 -- 172.21.15.77:0/590591933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b48634f0 msgr2=0x5584b4865980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64dca1b640 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b48634f0 0x5584b4865980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64dca1b640 1 -- 172.21.15.77:0/590591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 msgr2=0x5584b4862e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64dca1b640 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b4862e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64dca1b640 1 -- 172.21.15.77:0/590591933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584b459e7e0 con 0x5584b485d990 2024-03-16T09:43:00.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64d77fe640 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b4862e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:00.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64dca1b640 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b485ddb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f64d0034ec0 tx=0x7f64d002bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a799c8642401d203 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64be7fc640 1 -- 172.21.15.77:0/590591933 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0004b40 con 0x5584b485d990 2024-03-16T09:43:00.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64be7fc640 1 -- 172.21.15.77:0/590591933 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64d0004ce0 con 0x5584b485d990 2024-03-16T09:43:00.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.338+0000 7f64be7fc640 1 -- 172.21.15.77:0/590591933 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0033e30 con 0x5584b485d990 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 -- 172.21.15.77:0/590591933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 msgr2=0x5584b485ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b485ddb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f64d0034ec0 tx=0x7f64d002bef0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 -- 172.21.15.77:0/590591933 shutdown_connections 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b48634f0 0x5584b4865980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b4862e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 --2- 172.21.15.77:0/590591933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b485ddb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 -- 172.21.15.77:0/590591933 >> 172.21.15.77:0/590591933 conn(0x5584b4825c50 msgr2=0x5584b4859300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 -- 172.21.15.77:0/590591933 shutdown_connections 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 -- 172.21.15.77:0/590591933 wait complete. 2024-03-16T09:43:00.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 -- start start 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 Processor -- start 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b4867ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.339+0000 7f64e1069740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b48683e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64e1069740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b48634f0 0x5584b486f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64e1069740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5584b4866440 con 0x5584b48634f0 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64e1069740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5584b477e0d0 con 0x5584b485d990 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64e1069740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5584b48665c0 con 0x5584b485e380 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64e1069740 10 client.0 _ll_register_callbacks cb 0x5584b4747760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64dca1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b48634f0 0x5584b486f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64dca1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b48634f0 0x5584b486f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43360/0 (socket says 172.21.15.77:43360) 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64dca1b640 1 -- 172.21.15.77:0/2412873148 learned_addr learned my addr 172.21.15.77:0/2412873148 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64d77fe640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b48683e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64dca1b640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b4867ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64d77fe640 1 -- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 msgr2=0x5584b4867ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64d77fe640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b4867ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64d77fe640 1 -- 172.21.15.77:0/2412873148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b48634f0 msgr2=0x5584b486f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64d77fe640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b48634f0 0x5584b486f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64d77fe640 1 -- 172.21.15.77:0/2412873148 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584b45fef70 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64dca1b640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b48634f0 0x5584b486f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64d77fe640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b48683e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5584b485e780 tx=0x7f64cc012b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9081918312ec78a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.340+0000 7f64b77fe640 1 -- 172.21.15.77:0/2412873148 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc004050 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64d0004ce0 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64b77fe640 1 -- 172.21.15.77:0/2412873148 <== mon.1 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f64cc0041f0 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64b77fe640 1 -- 172.21.15.77:0/2412873148 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f64cc0043f0 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 10 client.25489 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5584b4496170 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64b77fe640 1 -- 172.21.15.77:0/2412873148 <== mon.1 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64d0004ce0 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64b77fe640 1 -- 172.21.15.77:0/2412873148 <== mon.1 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f64cc003b50 con 0x5584b485e380 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 10 client.25489 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:00.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 10 client.25489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 20 client.25489 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b4873a10 0x5584b4875e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 4 client.25489 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584b47cf610 con 0x5584b4873a10 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.341+0000 7f64dca1b640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b4873a10 0x5584b4875e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.342+0000 7f64b77fe640 10 client.25489 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.342+0000 7f64dca1b640 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b4873a10 0x5584b4875e50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f38332ad06def8ed server_cookie=6340feaff31f5cee in_seq=0 out_seq=0 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.342+0000 7f64b77fe640 1 -- 172.21.15.77:0/2412873148 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5584b47cf610 con 0x5584b4873a10 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.342+0000 7f64b77fe640 10 client.25489 handle_command_reply: tid=0 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.342+0000 7f64e1069740 1 client.25489 shutdown 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.342+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 msgr2=0x5584b48683e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.342+0000 7f64e1069740 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b48683e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5584b485e780 tx=0x7f64cc012b40 comp rx=0 tx=0).stop 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 shutdown_connections 2024-03-16T09:43:00.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b4873a10 0x5584b4875e50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b48634f0 0x5584b486f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b48683e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64b77fe640 0 client.25489 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 --2- 172.21.15.77:0/2412873148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b4867ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 >> 172.21.15.77:0/2412873148 conn(0x5584b4825c50 msgr2=0x5584b48584e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 shutdown_connections 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 1 -- 172.21.15.77:0/2412873148 wait complete. 2024-03-16T09:43:00.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.343+0000 7f64e1069740 20 client.25489 trim_cache size 0 max 16384 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 Processor -- start 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 -- start start 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b4825c50 0x5584b4857c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b486f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b4858190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f64d81a92d0 con 0x5584b4825c50 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f64d81a6ba0 con 0x5584b485d990 2024-03-16T09:43:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64e1069740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64d81a4a10 con 0x5584b485e380 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64dca1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b4825c50 0x5584b4857c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64dca1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b4825c50 0x5584b4857c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43364/0 (socket says 172.21.15.77:43364) 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.360+0000 7f64dca1b640 1 -- 172.21.15.77:0/1882596192 learned_addr learned my addr 172.21.15.77:0/1882596192 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b4858190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64d77fe640 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b486f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64dca1b640 1 -- 172.21.15.77:0/1882596192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 msgr2=0x5584b4858190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b4858190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64dca1b640 1 -- 172.21.15.77:0/1882596192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 msgr2=0x5584b486f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b486f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64dca1b640 1 -- 172.21.15.77:0/1882596192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584b4868fd0 con 0x5584b4825c50 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b4825c50 0x5584b4857c50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f64d0036010 tx=0x7f64d0034f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7f553e0f70c133 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64bdffb640 1 -- 172.21.15.77:0/1882596192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0064070 con 0x5584b4825c50 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64bdffb640 1 -- 172.21.15.77:0/1882596192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64d0004c40 con 0x5584b4825c50 2024-03-16T09:43:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.361+0000 7f64bdffb640 1 -- 172.21.15.77:0/1882596192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0003bd0 con 0x5584b4825c50 2024-03-16T09:43:00.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 -- 172.21.15.77:0/1882596192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b4825c50 msgr2=0x5584b4857c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b4825c50 0x5584b4857c50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f64d0036010 tx=0x7f64d0034f90 comp rx=0 tx=0).stop 2024-03-16T09:43:00.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 -- 172.21.15.77:0/1882596192 shutdown_connections 2024-03-16T09:43:00.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485e380 0x5584b4858190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b485d990 0x5584b486f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 --2- 172.21.15.77:0/1882596192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b4825c50 0x5584b4857c50 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 -- 172.21.15.77:0/1882596192 >> 172.21.15.77:0/1882596192 conn(0x5584b4867a60 msgr2=0x5584b485f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 -- 172.21.15.77:0/1882596192 shutdown_connections 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 -- 172.21.15.77:0/1882596192 wait complete. 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 -- start start 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.362+0000 7f64e1069740 1 Processor -- start 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64e1069740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b4825c50 0x5584b486d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64e1069740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485d990 0x5584b486d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64e1069740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b486e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64e1069740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f64d81a5060 con 0x5584b485e380 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64e1069740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f64d81a64d0 con 0x5584b4825c50 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64e1069740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64d81a5730 con 0x5584b485d990 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64d77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485d990 0x5584b486d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64d77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485d990 0x5584b486d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37486/0 (socket says 172.21.15.77:37486) 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64d77fe640 1 -- 172.21.15.77:0/1901001493 learned_addr learned my addr 172.21.15.77:0/1901001493 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b486e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b4825c50 0x5584b486d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64e1069740 10 client.0 _ll_register_callbacks cb 0x7f64cc009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64dca1b640 1 -- 172.21.15.77:0/1901001493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485d990 msgr2=0x5584b486d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485d990 0x5584b486d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64dca1b640 1 -- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b4825c50 msgr2=0x5584b486d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.363+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b4825c50 0x5584b486d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64dca1b640 1 -- 172.21.15.77:0/1901001493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584b486edc0 con 0x5584b485e380 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64d77fe640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485d990 0x5584b486d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64dca1b640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b486e7d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f64d0056040 tx=0x7f64d0004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52b96680c946f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64b77fe640 1 -- 172.21.15.77:0/1901001493 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0064040 con 0x5584b485e380 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64b77fe640 1 -- 172.21.15.77:0/1901001493 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f64d0042c50 con 0x5584b485e380 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64d0004c40 con 0x5584b485e380 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64b77fe640 1 -- 172.21.15.77:0/1901001493 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f64d005fb30 con 0x5584b485e380 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64e1069740 10 client.15687 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:00.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5584b48da3f0 con 0x5584b485e380 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64b77fe640 1 -- 172.21.15.77:0/1901001493 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64d0004c40 con 0x5584b485e380 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64b77fe640 1 -- 172.21.15.77:0/1901001493 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f64d0074020 con 0x5584b485e380 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64e1069740 10 client.15687 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64e1069740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.364+0000 7f64e1069740 20 client.15687 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.365+0000 7f64e1069740 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b48634f0 0x5584b48dc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.365+0000 7f64e1069740 4 client.15687 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.365+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5584b4859650 con 0x5584b48634f0 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.365+0000 7f64d77fe640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b48634f0 0x5584b48dc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.365+0000 7f64d77fe640 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b48634f0 0x5584b48dc8b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf75e0c94a1d7c0a server_cookie=19d07f47a766fef2 in_seq=0 out_seq=0 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.365+0000 7f64b77fe640 10 client.15687 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.366+0000 7f64b77fe640 1 -- 172.21.15.77:0/1901001493 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5584b4859650 con 0x5584b48634f0 2024-03-16T09:43:00.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.366+0000 7f64b77fe640 10 client.15687 handle_command_reply: tid=0 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.366+0000 7f64e1069740 1 client.15687 shutdown 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.366+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 msgr2=0x5584b486e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.366+0000 7f64e1069740 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b486e7d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f64d0056040 tx=0x7f64d0004df0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 shutdown_connections 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5584b48634f0 0x5584b48dc8b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5584b485e380 0x5584b486e7d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64b77fe640 0 client.15687 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584b485d990 0x5584b486d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 --2- 172.21.15.77:0/1901001493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584b4825c50 0x5584b486d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 >> 172.21.15.77:0/1901001493 conn(0x5584b4867a60 msgr2=0x5584b485c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 shutdown_connections 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 1 -- 172.21.15.77:0/1901001493 wait complete. 2024-03-16T09:43:00.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64e1069740 20 client.15687 trim_cache size 0 max 16384 2024-03-16T09:43:00.375 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:43:00.375 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T09:43:00.375 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "cd9538e8-ef53-4bcc-905c-53cf2073a6b8", 2024-03-16T09:43:00.375 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T09:43:00.375 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f64c00a2880 msgr2=0x7f64c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.367+0000 7f64de47e640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f64c00a2880 0x7f64c00a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f64c809b410 tx=0x7f64c8018420 comp rx=0 tx=0).stop 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d81006e0 msgr2=0x7f64d810b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d81006e0 0x7f64d810b060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64c800e9a0 tx=0x7f64c800ee60 comp rx=0 tx=0).stop 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 shutdown_connections 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f64c00a2880 0x7f64c00a4d40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64d8102290 0x7f64d8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64d81018d0 0x7f64d81096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 --2- 172.21.15.77:0/3300876233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64d81006e0 0x7f64d810b060 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 >> 172.21.15.77:0/3300876233 conn(0x7f64d80fbc70 msgr2=0x7f64d8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 shutdown_connections 2024-03-16T09:43:00.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.368+0000 7f64de47e640 1 -- 172.21.15.77:0/3300876233 wait complete. 2024-03-16T09:43:00.384 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:43:00.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:43:00.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 -- 172.21.15.77:0/90059619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1018d0 msgr2=0x7f92dc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 --2- 172.21.15.77:0/90059619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1018d0 0x7f92dc101d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009860 tx=0x7f92d002bb70 comp rx=0 tx=0).stop 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 -- 172.21.15.77:0/90059619 shutdown_connections 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 --2- 172.21.15.77:0/90059619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92dc102290 0x7f92dc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 --2- 172.21.15.77:0/90059619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1018d0 0x7f92dc101d50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 --2- 172.21.15.77:0/90059619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1006e0 0x7f92dc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 -- 172.21.15.77:0/90059619 >> 172.21.15.77:0/90059619 conn(0x7f92dc0fbc70 msgr2=0x7f92dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.486+0000 7f92e0c75640 1 -- 172.21.15.77:0/90059619 shutdown_connections 2024-03-16T09:43:00.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.487+0000 7f92e0c75640 1 -- 172.21.15.77:0/90059619 wait complete. 2024-03-16T09:43:00.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.487+0000 7f92e0c75640 1 Processor -- start 2024-03-16T09:43:00.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.487+0000 7f92e0c75640 1 -- start start 2024-03-16T09:43:00.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92e0c75640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1006e0 0x7f92dc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92e0c75640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1018d0 0x7f92dc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92e0c75640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92dc102290 0x7f92dc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92e0c75640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10c210 con 0x7f92dc1018d0 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92e0c75640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10c090 con 0x7f92dc102290 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92e0c75640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10c390 con 0x7f92dc1006e0 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1018d0 0x7f92dc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1018d0 0x7f92dc197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43392/0 (socket says 172.21.15.77:43392) 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 -- 172.21.15.77:0/1450950806 learned_addr learned my addr 172.21.15.77:0/1450950806 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92da59c640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1006e0 0x7f92dc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92dad9d640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92dc102290 0x7f92dc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 -- 172.21.15.77:0/1450950806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1006e0 msgr2=0x7f92dc197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1006e0 0x7f92dc197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 -- 172.21.15.77:0/1450950806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92dc102290 msgr2=0x7f92dc19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92dc102290 0x7f92dc19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.488+0000 7f92d9d9b640 1 -- 172.21.15.77:0/1450950806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc19f6b0 con 0x7f92dc1018d0 2024-03-16T09:43:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.489+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1018d0 0x7f92dc197fd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f92d00057c0 tx=0x7f92d00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.489+0000 7f92cb7fe640 1 -- 172.21.15.77:0/1450950806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0041030 con 0x7f92dc1018d0 2024-03-16T09:43:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.489+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92dc19f940 con 0x7f92dc1018d0 2024-03-16T09:43:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.489+0000 7f92cb7fe640 1 -- 172.21.15.77:0/1450950806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92d002c9c0 con 0x7f92dc1018d0 2024-03-16T09:43:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.489+0000 7f92cb7fe640 1 -- 172.21.15.77:0/1450950806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0039040 con 0x7f92dc1018d0 2024-03-16T09:43:00.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.489+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92dc19fe30 con 0x7f92dc1018d0 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 Processor -- start 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 -- start start 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46004590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46009660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46009cd0 0x55da4600c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da45f69500 con 0x55da46004170 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55da45f11be0 con 0x55da46009cd0 2024-03-16T09:43:00.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92e385b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da45f655a0 con 0x55da46004b60 2024-03-16T09:43:00.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92dad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46004590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92dad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46004590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43404/0 (socket says 172.21.15.77:43404) 2024-03-16T09:43:00.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92dad9d640 1 -- 172.21.15.77:0/1840323607 learned_addr learned my addr 172.21.15.77:0/1840323607 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.492+0000 7f92da59c640 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46009660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92dad9d640 1 -- 172.21.15.77:0/1840323607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 msgr2=0x55da46009660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92dad9d640 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46009660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92dad9d640 1 -- 172.21.15.77:0/1840323607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46009cd0 msgr2=0x55da4600c160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T09:43:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92dad9d640 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46009cd0 0x55da4600c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92dad9d640 1 -- 172.21.15.77:0/1840323607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da45f254b0 con 0x55da46004170 2024-03-16T09:43:00.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92dad9d640 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46004590 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f92d400e580 tx=0x7f92d40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60202dafc0afae13 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92c8ff9640 1 -- 172.21.15.77:0/1840323607 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4003970 con 0x55da46004170 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92c8ff9640 1 -- 172.21.15.77:0/1840323607 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92d400d830 con 0x55da46004170 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92c8ff9640 1 -- 172.21.15.77:0/1840323607 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4004490 con 0x55da46004170 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.493+0000 7f92cb7fe640 1 -- 172.21.15.77:0/1450950806 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f92d0044070 con 0x7f92dc1018d0 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92cb7fe640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f92c40a2800 0x7f92c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92da59c640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f92c40a2800 0x7f92c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92cb7fe640 1 -- 172.21.15.77:0/1450950806 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f92d00effd0 con 0x7f92dc1018d0 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92e385b740 1 -- 172.21.15.77:0/1840323607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 msgr2=0x55da46004590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92e385b740 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46004590 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f92d400e580 tx=0x7f92d40095c0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92e385b740 1 -- 172.21.15.77:0/1840323607 shutdown_connections 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92e385b740 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46009cd0 0x55da4600c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92e385b740 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46009660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92e385b740 1 --2- 172.21.15.77:0/1840323607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46004590 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.494+0000 7f92e385b740 1 -- 172.21.15.77:0/1840323607 >> 172.21.15.77:0/1840323607 conn(0x55da45fff6e0 msgr2=0x55da45fffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92da59c640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f92c40a2800 0x7f92c40a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55da46004f60 tx=0x7f92cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 -- 172.21.15.77:0/1840323607 shutdown_connections 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 -- 172.21.15.77:0/1840323607 wait complete. 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 -- start start 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 Processor -- start 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da4600e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46009cd0 0x55da46015c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da4600cc80 con 0x55da46009cd0 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55da45f25160 con 0x55da46004b60 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.495+0000 7f92e385b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da4600ce00 con 0x55da46004170 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92e385b740 10 client.0 _ll_register_callbacks cb 0x55da45f09150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da4600e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da4600e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:37518/0 (socket says 172.21.15.77:37518) 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 -- 172.21.15.77:0/679502369 learned_addr learned my addr 172.21.15.77:0/679502369 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92dad9d640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46009cd0 0x55da46015c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 -- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 msgr2=0x55da4600ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 -- 172.21.15.77:0/679502369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46009cd0 msgr2=0x55da46015c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46009cd0 0x55da46015c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 -- 172.21.15.77:0/679502369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da45d477d0 con 0x55da46004170 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92dad9d640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46009cd0 0x55da46015c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92da59c640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da4600e700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f92cc013010 tx=0x7f92cc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59949ce7f141758c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92c1ffb640 1 -- 172.21.15.77:0/679502369 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc02d060 con 0x55da46004170 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92c1ffb640 1 -- 172.21.15.77:0/679502369 <== mon.1 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f92cc018030 con 0x55da46004170 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92d400d830 con 0x55da46004170 2024-03-16T09:43:00.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92c1ffb640 1 -- 172.21.15.77:0/679502369 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f92cc011850 con 0x55da46004170 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92e385b740 10 client.25507 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.496+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55da45c7d3f0 con 0x55da46004170 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92c1ffb640 1 -- 172.21.15.77:0/679502369 <== mon.1 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92d400d830 con 0x55da46004170 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92c1ffb640 1 -- 172.21.15.77:0/679502369 <== mon.1 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f92cc028af0 con 0x55da46004170 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92e385b740 10 client.25507 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92e385b740 10 client.25507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92e385b740 20 client.25507 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92e385b740 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da4601a260 0x55da4601c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92e385b740 4 client.25507 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da45f76a40 con 0x55da4601a260 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da4601a260 0x55da4601c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.497+0000 7f92c1ffb640 10 client.25507 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da4601a260 0x55da4601c6e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64bdc45cba57c7d1 server_cookie=df27011076d535a8 in_seq=0 out_seq=0 2024-03-16T09:43:00.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92c1ffb640 1 -- 172.21.15.77:0/679502369 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da45f76a40 con 0x55da4601a260 2024-03-16T09:43:00.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92c1ffb640 10 client.25507 handle_command_reply: tid=0 2024-03-16T09:43:00.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 client.25507 shutdown 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 msgr2=0x55da4600e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da4600e700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f92cc013010 tx=0x7f92cc004730 comp rx=0 tx=0).stop 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 shutdown_connections 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da4601a260 0x55da4601c6e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46009cd0 0x55da46015c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92c1ffb640 0 client.25507 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 --2- 172.21.15.77:0/679502369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da4600e700 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 >> 172.21.15.77:0/679502369 conn(0x55da45fff6e0 msgr2=0x55da45ffe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.498+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 shutdown_connections 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.499+0000 7f92e385b740 1 -- 172.21.15.77:0/679502369 wait complete. 2024-03-16T09:43:00.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.499+0000 7f92e385b740 20 client.25507 trim_cache size 0 max 16384 2024-03-16T09:43:00.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 Processor -- start 2024-03-16T09:43:00.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 -- start start 2024-03-16T09:43:00.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da45fff6e0 0x55da45fffac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da46015c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc1ab530 con 0x55da45fff6e0 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc1a8e00 con 0x55da46004b60 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92e385b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc1a6c50 con 0x55da46004170 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92da59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da45fff6e0 0x55da45fffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92dad9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92dad9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50418/0 (socket says 172.21.15.77:50418) 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92dad9d640 1 -- 172.21.15.77:0/2474436489 learned_addr learned my addr 172.21.15.77:0/2474436489 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.533+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da46015c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92dad9d640 1 -- 172.21.15.77:0/2474436489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 msgr2=0x55da46015c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92dad9d640 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da46015c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92dad9d640 1 -- 172.21.15.77:0/2474436489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da45fff6e0 msgr2=0x55da45fffac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92dad9d640 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da45fff6e0 0x55da45fffac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92dad9d640 1 -- 172.21.15.77:0/2474436489 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da45c3d0f0 con 0x55da46004b60 2024-03-16T09:43:00.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92dad9d640 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ea20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f92d400e520 tx=0x7f92d4009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f4576dd15e81642 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92c3fff640 1 -- 172.21.15.77:0/2474436489 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4003a70 con 0x55da46004b60 2024-03-16T09:43:00.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92c3fff640 1 -- 172.21.15.77:0/2474436489 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92d402a070 con 0x55da46004b60 2024-03-16T09:43:00.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.534+0000 7f92c3fff640 1 -- 172.21.15.77:0/2474436489 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4026ba0 con 0x55da46004b60 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 -- 172.21.15.77:0/2474436489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 msgr2=0x55da4600ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ea20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f92d400e520 tx=0x7f92d4009da0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 -- 172.21.15.77:0/2474436489 shutdown_connections 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da46004b60 0x55da4600ea20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004170 0x55da46015c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 --2- 172.21.15.77:0/2474436489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da45fff6e0 0x55da45fffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 -- 172.21.15.77:0/2474436489 >> 172.21.15.77:0/2474436489 conn(0x55da4600d910 msgr2=0x55da4600e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 -- 172.21.15.77:0/2474436489 shutdown_connections 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 -- 172.21.15.77:0/2474436489 wait complete. 2024-03-16T09:43:00.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 -- start start 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.535+0000 7f92e385b740 1 Processor -- start 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92e385b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da45fff6e0 0x55da46013230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92e385b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46013770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92e385b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46014730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92e385b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc1a72c0 con 0x55da46004170 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92e385b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc1a8730 con 0x55da45fff6e0 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92e385b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc1a7990 con 0x55da46004b60 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da45fff6e0 0x55da46013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92e385b740 10 client.0 _ll_register_callbacks cb 0x7f92cc009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da45fff6e0 0x55da46013230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50434/0 (socket says 172.21.15.77:50434) 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 -- 172.21.15.77:0/2647320210 learned_addr learned my addr 172.21.15.77:0/2647320210 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46013770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92dad9d640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46014730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 -- 172.21.15.77:0/2647320210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 msgr2=0x55da46014730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46014730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 -- 172.21.15.77:0/2647320210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 msgr2=0x55da46013770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46013770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 -- 172.21.15.77:0/2647320210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da45f309d0 con 0x55da45fff6e0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92dad9d640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46014730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46013770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92da59c640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da45fff6e0 0x55da46013230 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000c00 tx=0x7f92cc011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45ac11deb0e353e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.536+0000 7f92c1ffb640 1 -- 172.21.15.77:0/2647320210 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc038070 con 0x55da45fff6e0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92d402a070 con 0x55da45fff6e0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92c1ffb640 1 -- 172.21.15.77:0/2647320210 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f92cc0287f0 con 0x55da45fff6e0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92c1ffb640 1 -- 172.21.15.77:0/2647320210 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f92cc044050 con 0x55da45fff6e0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 10 client.25583 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55da46015ed0 con 0x55da45fff6e0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92c1ffb640 1 -- 172.21.15.77:0/2647320210 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92cc0289f0 con 0x55da45fff6e0 2024-03-16T09:43:00.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92c1ffb640 1 -- 172.21.15.77:0/2647320210 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f92cc034450 con 0x55da45fff6e0 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 10 client.25583 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 10 client.25583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 20 client.25583 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da46009cd0 0x55da46082c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 4 client.25583 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f92d402a070 con 0x55da46009cd0 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.537+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da46009cd0 0x55da46082c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92c1ffb640 10 client.25583 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92d9d9b640 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da46009cd0 0x55da46082c40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=827abdc40698780a server_cookie=1bc6552484e6a831 in_seq=0 out_seq=0 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92c1ffb640 1 -- 172.21.15.77:0/2647320210 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f92d402a070 con 0x55da46009cd0 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92c1ffb640 10 client.25583 handle_command_reply: tid=0 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 client.25583 shutdown 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da45fff6e0 msgr2=0x55da46013230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da45fff6e0 0x55da46013230 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000c00 tx=0x7f92cc011d10 comp rx=0 tx=0).stop 2024-03-16T09:43:00.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 shutdown_connections 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55da46009cd0 0x55da46082c40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da46004b60 0x55da46014730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da46004170 0x55da46013770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92c1ffb640 0 client.25583 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 --2- 172.21.15.77:0/2647320210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da45fff6e0 0x55da46013230 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 >> 172.21.15.77:0/2647320210 conn(0x55da4600d910 msgr2=0x55da46002940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 shutdown_connections 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 1 -- 172.21.15.77:0/2647320210 wait complete. 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.538+0000 7f92e385b740 20 client.25583 trim_cache size 0 max 16384 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T09:43:00.547 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T09:43:00.548 INFO:teuthology.orchestra.run.smithi077.stdout: "cd9538e8-ef53-4bcc-905c-53cf2073a6b8": { 2024-03-16T09:43:00.548 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T09:43:00.548 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "cd9538e8-ef53-4bcc-905c-53cf2073a6b8", 2024-03-16T09:43:00.548 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T09:43:00.548 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:43:00.548 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:43:00.548 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f92c40a2800 msgr2=0x7f92c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f92c40a2800 0x7f92c40a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55da46004f60 tx=0x7f92cc005f90 comp rx=0 tx=0).stop 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1018d0 msgr2=0x7f92dc197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1018d0 0x7f92dc197fd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f92d00057c0 tx=0x7f92d00042f0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 shutdown_connections 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f92c40a2800 0x7f92c40a4cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92dc102290 0x7f92dc19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc1018d0 0x7f92dc197fd0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 --2- 172.21.15.77:0/1450950806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92dc1006e0 0x7f92dc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 >> 172.21.15.77:0/1450950806 conn(0x7f92dc0fbc70 msgr2=0x7f92dc0689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 shutdown_connections 2024-03-16T09:43:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:00.539+0000 7f92e0c75640 1 -- 172.21.15.77:0/1450950806 wait complete. 2024-03-16T09:43:00.553 INFO:tasks.cephfs.filesystem:scrub status for tag:cd9538e8-ef53-4bcc-905c-53cf2073a6b8 - {'path': '/', 'tag': 'cd9538e8-ef53-4bcc-905c-53cf2073a6b8', 'options': 'recursive,force'} 2024-03-16T09:43:00.913 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:00 smithi077 ceph-mon[32171]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:43:00.913 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:00 smithi077 ceph-mon[32171]: scrub queued for path: / 2024-03-16T09:43:00.913 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:00 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:00.913 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:00 smithi077 ceph-mon[32171]: scrub summary: active paths [/] 2024-03-16T09:43:00.913 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:00 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:00.914 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:00 smithi077 ceph-mon[32171]: scrub summary: idle 2024-03-16T09:43:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:00 smithi119 ceph-mon[35632]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:43:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:00 smithi119 ceph-mon[35632]: scrub queued for path: / 2024-03-16T09:43:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:00 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:00 smithi119 ceph-mon[35632]: scrub summary: active paths [/] 2024-03-16T09:43:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:00 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:00 smithi119 ceph-mon[35632]: scrub summary: idle 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 -- 172.21.15.77:0/1023168262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 msgr2=0x7f59cc100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 --2- 172.21.15.77:0/1023168262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 0x7f59cc100e70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009900 tx=0x7f59c002bc80 comp rx=0 tx=0).stop 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 -- 172.21.15.77:0/1023168262 shutdown_connections 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 --2- 172.21.15.77:0/1023168262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59cc1013b0 0x7f59cc101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 --2- 172.21.15.77:0/1023168262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 0x7f59cc100e70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 --2- 172.21.15.77:0/1023168262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 0x7f59cc1004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 -- 172.21.15.77:0/1023168262 >> 172.21.15.77:0/1023168262 conn(0x7f59cc0fbc70 msgr2=0x7f59cc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:01.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 -- 172.21.15.77:0/1023168262 shutdown_connections 2024-03-16T09:43:01.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.279+0000 7f59d2436640 1 -- 172.21.15.77:0/1023168262 wait complete. 2024-03-16T09:43:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.280+0000 7f59d2436640 1 Processor -- start 2024-03-16T09:43:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.280+0000 7f59d2436640 1 -- start start 2024-03-16T09:43:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59d2436640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 0x7f59cc199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59d2436640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 0x7f59cc19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59d2436640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59cc1013b0 0x7f59cc1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59d2436640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc10f580 con 0x7f59cc100a10 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59cb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 0x7f59cc19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59cbfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 0x7f59cc199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59d2436640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc10f400 con 0x7f59cc1000b0 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59cbfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 0x7f59cc199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:50454/0 (socket says 172.21.15.77:50454) 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59cbfff640 1 -- 172.21.15.77:0/3725047125 learned_addr learned my addr 172.21.15.77:0/3725047125 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc10f700 con 0x7f59cc1013b0 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.281+0000 7f59d09d3640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59cc1013b0 0x7f59cc1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59cbfff640 1 -- 172.21.15.77:0/3725047125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59cc1013b0 msgr2=0x7f59cc1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59cbfff640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59cc1013b0 0x7f59cc1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59cbfff640 1 -- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 msgr2=0x7f59cc19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59cbfff640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 0x7f59cc19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59cbfff640 1 -- 172.21.15.77:0/3725047125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc1a17b0 con 0x7f59cc1000b0 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 0x7f59cc19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59cbfff640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 0x7f59cc199b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00e9f0 tx=0x7f59bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59c97fa640 1 -- 172.21.15.77:0/3725047125 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc00cd80 con 0x7f59cc1000b0 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59c97fa640 1 -- 172.21.15.77:0/3725047125 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59bc0108e0 con 0x7f59cc1000b0 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59cc1a1aa0 con 0x7f59cc1000b0 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59c97fa640 1 -- 172.21.15.77:0/3725047125 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc018920 con 0x7f59cc1000b0 2024-03-16T09:43:01.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.282+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59cc1a1f00 con 0x7f59cc1000b0 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 Processor -- start 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 -- start start 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7ade6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adeb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adee330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560e7ad4ba40 con 0x560e7ade6340 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7ade6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560e7acf4920 con 0x560e7ade6d30 2024-03-16T09:43:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d501d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e7ad47ae0 con 0x560e7adebea0 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7ade6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43462/0 (socket says 172.21.15.77:43462) 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59d09d3640 1 -- 172.21.15.77:0/1707496609 learned_addr learned my addr 172.21.15.77:0/1707496609 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59c97fa640 1 -- 172.21.15.77:0/3725047125 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f59bc010a90 con 0x7f59cc1000b0 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.285+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59cbfff640 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adeb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59d09d3640 1 -- 172.21.15.77:0/1707496609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 msgr2=0x560e7adee330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59d09d3640 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adee330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59d09d3640 1 -- 172.21.15.77:0/1707496609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 msgr2=0x560e7adeb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59d09d3640 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adeb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59d09d3640 1 -- 172.21.15.77:0/1707496609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e7ad0c460 con 0x560e7ade6340 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59cbfff640 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adeb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:43:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59d09d3640 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7ade6760 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f59c4009fd0 tx=0x7f59c40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a472fc9fe0855a7a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59c97fa640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f59b40a2920 0x7f59b40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59b27fc640 1 -- 172.21.15.77:0/1707496609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c4004d20 con 0x560e7ade6340 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59cbfff640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f59b40a2920 0x7f59b40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59b27fc640 1 -- 172.21.15.77:0/1707496609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59c4004050 con 0x560e7ade6340 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59b27fc640 1 -- 172.21.15.77:0/1707496609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c4005280 con 0x560e7ade6340 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.286+0000 7f59c97fa640 1 -- 172.21.15.77:0/3725047125 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f59bc0ceda0 con 0x7f59cc1000b0 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 -- 172.21.15.77:0/1707496609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 msgr2=0x560e7ade6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7ade6760 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f59c4009fd0 tx=0x7f59c40118d0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59cbfff640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f59b40a2920 0x7f59b40a4de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f59bc09b4a0 tx=0x7f59bc06a6c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 -- 172.21.15.77:0/1707496609 shutdown_connections 2024-03-16T09:43:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adeb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 --2- 172.21.15.77:0/1707496609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7ade6760 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 -- 172.21.15.77:0/1707496609 >> 172.21.15.77:0/1707496609 conn(0x560e7ade18b0 msgr2=0x560e7ade1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 -- 172.21.15.77:0/1707496609 shutdown_connections 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.287+0000 7f59d501d740 1 -- 172.21.15.77:0/1707496609 wait complete. 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 -- start start 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 Processor -- start 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7adebea0 0x560e7adf7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560e7adeee50 con 0x560e7ade6340 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560e7ad07d70 con 0x560e7adebea0 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e7adeefd0 con 0x560e7ade6d30 2024-03-16T09:43:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d501d740 10 client.0 _ll_register_callbacks cb 0x560e7aa5fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43470/0 (socket says 172.21.15.77:43470) 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.288+0000 7f59d09d3640 1 -- 172.21.15.77:0/3987318238 learned_addr learned my addr 172.21.15.77:0/3987318238 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d09d3640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7adebea0 0x560e7adf7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d09d3640 1 -- 172.21.15.77:0/3987318238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 msgr2=0x560e7adf0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d09d3640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d09d3640 1 -- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7adebea0 msgr2=0x560e7adf7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d09d3640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7adebea0 0x560e7adf7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d09d3640 1 -- 172.21.15.77:0/3987318238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e7aa60210 con 0x560e7ade6340 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:43:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d09d3640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf06d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f59c4003fd0 tx=0x7f59c40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5262540aa16f05a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59ab7fe640 1 -- 172.21.15.77:0/3987318238 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c400f8f0 con 0x560e7ade6340 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59ab7fe640 1 -- 172.21.15.77:0/3987318238 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f59c400fa90 con 0x560e7ade6340 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59c4004050 con 0x560e7ade6340 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59ab7fe640 1 -- 172.21.15.77:0/3987318238 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f59c400fc90 con 0x560e7ade6340 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d501d740 10 client.15717 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.289+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560e7ab2c620 con 0x560e7ade6340 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59ab7fe640 1 -- 172.21.15.77:0/3987318238 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59c4004050 con 0x560e7ade6340 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59ab7fe640 1 -- 172.21.15.77:0/3987318238 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f59c4031400 con 0x560e7ade6340 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59d501d740 10 client.15717 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59d501d740 10 client.15717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59d501d740 20 client.15717 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59d501d740 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7adfc230 0x560e7adfe620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59d501d740 4 client.15717 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e7ad58f80 con 0x560e7adfc230 2024-03-16T09:43:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.290+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7adfc230 0x560e7adfe620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.291+0000 7f59ab7fe640 10 client.15717 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.291+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7adfc230 0x560e7adfe620 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12e0597f42a6725f server_cookie=5ec2a3457dd12a63 in_seq=0 out_seq=0 2024-03-16T09:43:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:00 smithi129 ceph-mon[35841]: pgmap v379: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T09:43:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:00 smithi129 ceph-mon[35841]: scrub queued for path: / 2024-03-16T09:43:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:00 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:00 smithi129 ceph-mon[35841]: scrub summary: active paths [/] 2024-03-16T09:43:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:00 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:00 smithi129 ceph-mon[35841]: scrub summary: idle 2024-03-16T09:43:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.291+0000 7f59ab7fe640 1 -- 172.21.15.77:0/3987318238 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e7ad58f80 con 0x560e7adfc230 2024-03-16T09:43:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.291+0000 7f59ab7fe640 10 client.15717 handle_command_reply: tid=0 2024-03-16T09:43:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 client.15717 shutdown 2024-03-16T09:43:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 msgr2=0x560e7adf06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf06d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f59c4003fd0 tx=0x7f59c40047d0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 shutdown_connections 2024-03-16T09:43:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7adfc230 0x560e7adfe620 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7adebea0 0x560e7adf7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 --2- 172.21.15.77:0/3987318238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf06d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59ab7fe640 0 client.15717 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 >> 172.21.15.77:0/3987318238 conn(0x560e7ade18b0 msgr2=0x560e7ade0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 shutdown_connections 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 1 -- 172.21.15.77:0/3987318238 wait complete. 2024-03-16T09:43:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.292+0000 7f59d501d740 20 client.15717 trim_cache size 0 max 16384 2024-03-16T09:43:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 Processor -- start 2024-03-16T09:43:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 -- start start 2024-03-16T09:43:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6340 0x560e7adf0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7adebea0 0x560e7adefab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc1ad660 con 0x560e7adebea0 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc1aaf30 con 0x560e7ade6340 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d501d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc1a8d80 con 0x560e7ade6d30 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7adebea0 0x560e7adefab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7adebea0 0x560e7adefab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43482/0 (socket says 172.21.15.77:43482) 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d09d3640 1 -- 172.21.15.77:0/3974687969 learned_addr learned my addr 172.21.15.77:0/3974687969 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.307+0000 7f59d09d3640 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6340 0x560e7adf0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59cb7fe640 1 -- 172.21.15.77:0/3974687969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6340 msgr2=0x560e7adf0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6340 0x560e7adf0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59cb7fe640 1 -- 172.21.15.77:0/3974687969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7adebea0 msgr2=0x560e7adefab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7adebea0 0x560e7adefab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59cb7fe640 1 -- 172.21.15.77:0/3974687969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e7add3140 con 0x560e7ade6d30 2024-03-16T09:43:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59d09d3640 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7adebea0 0x560e7adefab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf7c10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f59c002c1c0 tx=0x7f59c002c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a89f02569c2baaa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59b1ffb640 1 -- 172.21.15.77:0/3974687969 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c004d020 con 0x560e7ade6d30 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59b1ffb640 1 -- 172.21.15.77:0/3974687969 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59c0005040 con 0x560e7ade6d30 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.308+0000 7f59b1ffb640 1 -- 172.21.15.77:0/3974687969 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c00046c0 con 0x560e7ade6d30 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 -- 172.21.15.77:0/3974687969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 msgr2=0x560e7adf7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf7c10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f59c002c1c0 tx=0x7f59c002c7d0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 -- 172.21.15.77:0/3974687969 shutdown_connections 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7adebea0 0x560e7adefab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7ade6d30 0x560e7adf7c10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 --2- 172.21.15.77:0/3974687969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6340 0x560e7adf0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 -- 172.21.15.77:0/3974687969 >> 172.21.15.77:0/3974687969 conn(0x560e7ade18b0 msgr2=0x560e7adef3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 -- 172.21.15.77:0/3974687969 shutdown_connections 2024-03-16T09:43:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 -- 172.21.15.77:0/3974687969 wait complete. 2024-03-16T09:43:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 -- start start 2024-03-16T09:43:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.309+0000 7f59d501d740 1 Processor -- start 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d501d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d501d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adf55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d501d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adf65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d501d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc1a93f0 con 0x560e7ade6340 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d501d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc1aa860 con 0x560e7ade6d30 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d501d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc1a9ac0 con 0x560e7adebea0 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d501d740 10 client.0 _ll_register_callbacks cb 0x7f59c4009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43498/0 (socket says 172.21.15.77:43498) 2024-03-16T09:43:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 -- 172.21.15.77:0/850671243 learned_addr learned my addr 172.21.15.77:0/850671243 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adf55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 -- 172.21.15.77:0/850671243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 msgr2=0x560e7adf65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adf65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 -- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 msgr2=0x560e7adf55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adf55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 -- 172.21.15.77:0/850671243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e7adf6bd0 con 0x560e7ade6340 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adf55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59d09d3640 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf5090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f59c4031820 tx=0x7f59c4003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97482b55a90f451 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.310+0000 7f59ab7fe640 1 -- 172.21.15.77:0/850671243 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c4031600 con 0x560e7ade6340 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59ab7fe640 1 -- 172.21.15.77:0/850671243 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f59c4031a10 con 0x560e7ade6340 2024-03-16T09:43:01.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59c0005040 con 0x560e7ade6340 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59ab7fe640 1 -- 172.21.15.77:0/850671243 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f59c4031c10 con 0x560e7ade6340 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 10 client.15723 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560e7adf7e50 con 0x560e7ade6340 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59ab7fe640 1 -- 172.21.15.77:0/850671243 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59c0005040 con 0x560e7ade6340 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59ab7fe640 1 -- 172.21.15.77:0/850671243 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f59c4034a10 con 0x560e7ade6340 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 10 client.15723 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 20 client.15723 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7ae62bc0 0x560e7ae64fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 4 client.15723 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e7ad532d0 con 0x560e7ae62bc0 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.311+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7ae62bc0 0x560e7ae64fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:01.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.312+0000 7f59cb7fe640 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7ae62bc0 0x560e7ae64fe0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d946937debe38bc8 server_cookie=6866e64a559822b9 in_seq=0 out_seq=0 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.312+0000 7f59ab7fe640 10 client.15723 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59ab7fe640 1 -- 172.21.15.77:0/850671243 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e7ad532d0 con 0x560e7ae62bc0 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59ab7fe640 10 client.15723 handle_command_reply: tid=0 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 client.15723 shutdown 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 msgr2=0x560e7adf5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf5090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f59c4031820 tx=0x7f59c4003fd0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 shutdown_connections 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560e7ae62bc0 0x560e7ae64fe0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e7adebea0 0x560e7adf65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59ab7fe640 0 client.15723 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560e7ade6d30 0x560e7adf55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 --2- 172.21.15.77:0/850671243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e7ade6340 0x560e7adf5090 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.315+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 >> 172.21.15.77:0/850671243 conn(0x560e7ade18b0 msgr2=0x560e7ade0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 shutdown_connections 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d501d740 1 -- 172.21.15.77:0/850671243 wait complete. 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d501d740 20 client.15723 trim_cache size 0 max 16384 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:43:01.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:43:01.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f59b40a2920 msgr2=0x7f59b40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f59b40a2920 0x7f59b40a4de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f59bc09b4a0 tx=0x7f59bc06a6c0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 msgr2=0x7f59cc199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:01.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 0x7f59cc199b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00e9f0 tx=0x7f59bc00eeb0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 shutdown_connections 2024-03-16T09:43:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f59b40a2920 0x7f59b40a4de0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59cc1013b0 0x7f59cc1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59cc100a10 0x7f59cc19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 --2- 172.21.15.77:0/3725047125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc1000b0 0x7f59cc199b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 >> 172.21.15.77:0/3725047125 conn(0x7f59cc0fbc70 msgr2=0x7f59cc0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 shutdown_connections 2024-03-16T09:43:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:01.316+0000 7f59d2436640 1 -- 172.21.15.77:0/3725047125 wait complete. 2024-03-16T09:43:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:02 smithi119 ceph-mon[35632]: pgmap v380: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:43:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:02 smithi077 ceph-mon[32171]: pgmap v380: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:43:03.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:02 smithi129 ceph-mon[35841]: pgmap v380: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:43:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:04 smithi119 ceph-mon[35632]: pgmap v381: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-16T09:43:05.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:04 smithi077 ceph-mon[32171]: pgmap v381: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-16T09:43:05.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:04 smithi129 ceph-mon[35841]: pgmap v381: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-16T09:43:07.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:06 smithi119 ceph-mon[35632]: pgmap v382: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T09:43:07.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:06 smithi077 ceph-mon[32171]: pgmap v382: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T09:43:07.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:06 smithi129 ceph-mon[35841]: pgmap v382: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T09:43:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:08 smithi119 ceph-mon[35632]: pgmap v383: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:43:09.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:08 smithi077 ceph-mon[32171]: pgmap v383: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:43:09.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:08 smithi129 ceph-mon[35841]: pgmap v383: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T09:43:11.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:10 smithi119 ceph-mon[35632]: pgmap v384: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:43:11.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:10 smithi077 ceph-mon[32171]: pgmap v384: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:43:11.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:10 smithi129 ceph-mon[35841]: pgmap v384: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:43:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:12 smithi119 ceph-mon[35632]: pgmap v385: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T09:43:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:12 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:43:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:12 smithi077 ceph-mon[32171]: pgmap v385: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T09:43:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:12 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:43:13.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:12 smithi129 ceph-mon[35841]: pgmap v385: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T09:43:13.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:12 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:43:13.548 INFO:tasks.workunit.client.0.smithi077.stderr:Updating files: 97% (11905/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr:Note: switching to '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'. 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr:state without impacting any branches by switching back to a branch. 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-16T09:43:14.185 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr: git switch -c 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr:Or undo this operation with: 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr: git switch - 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T09:43:14.186 INFO:tasks.workunit.client.0.smithi077.stderr:HEAD is now at 3d78f5ba07b Merge PR #54581 into wip-batrick-testing-20240314.183243 2024-03-16T09:43:14.189 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-16T09:43:14.253 INFO:tasks.workunit.client.0.smithi077.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-16T09:43:14.255 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T09:43:14.255 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 -- 172.21.15.77:0/364022641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 msgr2=0x7fee44108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 --2- 172.21.15.77:0/364022641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee44108b40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009ca0 tx=0x7fee3c02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 -- 172.21.15.77:0/364022641 shutdown_connections 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 --2- 172.21.15.77:0/364022641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee44108b40 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 --2- 172.21.15.77:0/364022641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee441018f0 0x7fee44101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 --2- 172.21.15.77:0/364022641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee44100700 0x7fee44100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 -- 172.21.15.77:0/364022641 >> 172.21.15.77:0/364022641 conn(0x7fee440fbc70 msgr2=0x7fee440fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 -- 172.21.15.77:0/364022641 shutdown_connections 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 -- 172.21.15.77:0/364022641 wait complete. 2024-03-16T09:43:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 Processor -- start 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.406+0000 7fee4b00b640 1 -- start start 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee4b00b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee44100700 0x7fee44199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee4b00b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee441018f0 0x7fee4419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee4b00b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee441a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee4b00b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee4410c230 con 0x7fee441022b0 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee4b00b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fee4410c0b0 con 0x7fee441018f0 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee4b00b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee4410c3b0 con 0x7fee44100700 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee441a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee441a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42138/0 (socket says 172.21.15.77:42138) 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 -- 172.21.15.77:0/2397070056 learned_addr learned my addr 172.21.15.77:0/2397070056 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee43fff640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee441018f0 0x7fee4419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee48da7640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee44100700 0x7fee44199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 -- 172.21.15.77:0/2397070056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee44100700 msgr2=0x7fee44199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee44100700 0x7fee44199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 -- 172.21.15.77:0/2397070056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee441018f0 msgr2=0x7fee4419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee441018f0 0x7fee4419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 -- 172.21.15.77:0/2397070056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee441a17c0 con 0x7fee441022b0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee495a8640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee441a1150 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee3c02c700 tx=0x7fee3c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee41ffb640 1 -- 172.21.15.77:0/2397070056 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c041020 con 0x7fee441022b0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee41ffb640 1 -- 172.21.15.77:0/2397070056 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee3c003e20 con 0x7fee441022b0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee41ffb640 1 -- 172.21.15.77:0/2397070056 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c039040 con 0x7fee441022b0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.407+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee441a1a50 con 0x7fee441022b0 2024-03-16T09:43:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.408+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee441a1f90 con 0x7fee441022b0 2024-03-16T09:43:14.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.410+0000 7fee41ffb640 1 -- 172.21.15.77:0/2397070056 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fee3c044020 con 0x7fee441022b0 2024-03-16T09:43:14.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.410+0000 7fee4dbf7740 1 Processor -- start 2024-03-16T09:43:14.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.410+0000 7fee4dbf7740 1 -- start start 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ea340 0x55f4837ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ead30 0x55f4837ef830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837f2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f48374fa40 con 0x55f4837ead30 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f4836f8920 con 0x55f4837efea0 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f48374bae0 con 0x55f4837ea340 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ead30 0x55f4837ef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee41ffb640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fee340a2920 0x7fee340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ead30 0x55f4837ef830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42146/0 (socket says 172.21.15.77:42146) 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 -- 172.21.15.77:0/442728384 learned_addr learned my addr 172.21.15.77:0/442728384 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee43fff640 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ea340 0x55f4837ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee495a8640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fee340a2920 0x7fee340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee43fff640 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 -- 172.21.15.77:0/442728384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ea340 msgr2=0x55f4837ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ea340 0x55f4837ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 -- 172.21.15.77:0/442728384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 msgr2=0x55f4837f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee41ffb640 1 -- 172.21.15.77:0/2397070056 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fee3c0f0b70 con 0x7fee441022b0 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 -- 172.21.15.77:0/442728384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f483710460 con 0x55f4837ead30 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee43fff640 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee48da7640 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ead30 0x55f4837ef830 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fee38009ca0 tx=0x7fee38011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450812e416016f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee495a8640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fee340a2920 0x7fee340a4de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0b9ef0 tx=0x7fee3c0a8040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee2effd640 1 -- 172.21.15.77:0/442728384 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38004220 con 0x55f4837ead30 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee2effd640 1 -- 172.21.15.77:0/442728384 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee380043c0 con 0x55f4837ead30 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.411+0000 7fee2effd640 1 -- 172.21.15.77:0/442728384 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee380198b0 con 0x55f4837ead30 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 -- 172.21.15.77:0/442728384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ead30 msgr2=0x55f4837ef830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ead30 0x55f4837ef830 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fee38009ca0 tx=0x7fee38011810 comp rx=0 tx=0).stop 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 -- 172.21.15.77:0/442728384 shutdown_connections 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ead30 0x55f4837ef830 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/442728384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ea340 0x55f4837ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 -- 172.21.15.77:0/442728384 >> 172.21.15.77:0/442728384 conn(0x55f4837e58b0 msgr2=0x55f4837e5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 -- 172.21.15.77:0/442728384 shutdown_connections 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 -- 172.21.15.77:0/442728384 wait complete. 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.412+0000 7fee4dbf7740 1 -- start start 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 1 Processor -- start 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ea340 0x55f4837f4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837efea0 0x55f4837fbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f4837f2e50 con 0x55f4837efea0 2024-03-16T09:43:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f48370bd70 con 0x55f4837ea340 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f4837f2fd0 con 0x55f4837ead30 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee4dbf7740 10 client.0 _ll_register_callbacks cb 0x55f483463a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837efea0 0x55f4837fbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837efea0 0x55f4837fbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42162/0 (socket says 172.21.15.77:42162) 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 -- 172.21.15.77:0/304212399 learned_addr learned my addr 172.21.15.77:0/304212399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee48da7640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ea340 0x55f4837f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 -- 172.21.15.77:0/304212399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 msgr2=0x55f4837f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 -- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ea340 msgr2=0x55f4837f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ea340 0x55f4837f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee43fff640 1 -- 172.21.15.77:0/304212399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f483464210 con 0x55f4837efea0 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.413+0000 7fee48da7640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee43fff640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837efea0 0x55f4837fbbd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee3000f460 tx=0x7fee30009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2460feea28e308 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee23fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee23fff640 1 -- 172.21.15.77:0/304212399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3000fea0 con 0x55f4837efea0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee23fff640 1 -- 172.21.15.77:0/304212399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fee30009040 con 0x55f4837efea0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee380043c0 con 0x55f4837efea0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee23fff640 1 -- 172.21.15.77:0/304212399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fee30004250 con 0x55f4837efea0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 10 client.15735 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f483530620 con 0x55f4837efea0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee23fff640 1 -- 172.21.15.77:0/304212399 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee380043c0 con 0x55f4837efea0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee23fff640 1 -- 172.21.15.77:0/304212399 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fee30020040 con 0x55f4837efea0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 10 client.15735 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 20 client.15735 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483800460 0x55f4838028d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 4 client.15735 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f48375cf80 con 0x55f483800460 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.414+0000 7fee48da7640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483800460 0x55f4838028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.415+0000 7fee48da7640 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483800460 0x55f4838028d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6e81629a27173e5 server_cookie=c18719a4932fee0d in_seq=0 out_seq=0 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.415+0000 7fee23fff640 10 client.15735 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.415+0000 7fee23fff640 1 -- 172.21.15.77:0/304212399 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f48375cf80 con 0x55f483800460 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.415+0000 7fee23fff640 10 client.15735 handle_command_reply: tid=0 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.415+0000 7fee4dbf7740 1 client.15735 shutdown 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837efea0 msgr2=0x55f4837fbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837efea0 0x55f4837fbbd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee3000f460 tx=0x7fee30009ac0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 shutdown_connections 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483800460 0x55f4838028d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837efea0 0x55f4837fbbd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee23fff640 0 client.15735 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/304212399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ea340 0x55f4837f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 >> 172.21.15.77:0/304212399 conn(0x55f4837e58b0 msgr2=0x55f4837e4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 shutdown_connections 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 1 -- 172.21.15.77:0/304212399 wait complete. 2024-03-16T09:43:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.416+0000 7fee4dbf7740 20 client.15735 trim_cache size 0 max 16384 2024-03-16T09:43:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.430+0000 7fee4dbf7740 1 Processor -- start 2024-03-16T09:43:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.430+0000 7fee4dbf7740 1 -- start start 2024-03-16T09:43:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.430+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837e9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.430+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ead30 0x55f4837fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.430+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837efea0 0x55f483807220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee441ad620 con 0x55f4837ea340 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fee441aaef0 con 0x55f4837ead30 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee441a8d40 con 0x55f4837efea0 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837e9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837e9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42168/0 (socket says 172.21.15.77:42168) 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 -- 172.21.15.77:0/2638155897 learned_addr learned my addr 172.21.15.77:0/2638155897 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee48da7640 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ead30 0x55f4837fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837efea0 0x55f483807220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 -- 172.21.15.77:0/2638155897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837efea0 msgr2=0x55f483807220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837efea0 0x55f483807220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 -- 172.21.15.77:0/2638155897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ead30 msgr2=0x55f4837fbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ead30 0x55f4837fbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 -- 172.21.15.77:0/2638155897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4837f3150 con 0x55f4837ea340 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee48da7640 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ead30 0x55f4837fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee43fff640 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837e9220 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fee30006010 tx=0x7fee30011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87be5457a264523 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee2e7fc640 1 -- 172.21.15.77:0/2638155897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30005130 con 0x55f4837ea340 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee2e7fc640 1 -- 172.21.15.77:0/2638155897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee30033040 con 0x55f4837ea340 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.431+0000 7fee2e7fc640 1 -- 172.21.15.77:0/2638155897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee300035f0 con 0x55f4837ea340 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2638155897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 msgr2=0x55f4837e9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837e9220 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fee30006010 tx=0x7fee30011ba0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2638155897 shutdown_connections 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837efea0 0x55f483807220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837ead30 0x55f4837fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2638155897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837e9220 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2638155897 >> 172.21.15.77:0/2638155897 conn(0x55f4837e58b0 msgr2=0x55f4837e8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2638155897 shutdown_connections 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2638155897 wait complete. 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 -- start start 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.432+0000 7fee4dbf7740 1 Processor -- start 2024-03-16T09:43:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837fa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837fa620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee441a93b0 con 0x55f4837ea340 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fee441aa820 con 0x55f4837efea0 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee441a9a80 con 0x55f4837ead30 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 10 client.0 _ll_register_callbacks cb 0x7fee30009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837fba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42170/0 (socket says 172.21.15.77:42170) 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 -- 172.21.15.77:0/2238110139 learned_addr learned my addr 172.21.15.77:0/2238110139 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee48da7640 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837fa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837fa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 -- 172.21.15.77:0/2238110139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 msgr2=0x55f4837fa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837fa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 -- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 msgr2=0x55f4837fa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837fa620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 -- 172.21.15.77:0/2238110139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4837fae30 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee43fff640 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837fba80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fee300099e0 tx=0x7fee30011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9732452bedefe95a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee23fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee23fff640 1 -- 172.21.15.77:0/2238110139 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3003f020 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee23fff640 1 -- 172.21.15.77:0/2238110139 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fee30005330 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.433+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee30033040 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee23fff640 1 -- 172.21.15.77:0/2238110139 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fee30005530 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 10 client.15741 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f4838678b0 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee23fff640 1 -- 172.21.15.77:0/2238110139 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee30033040 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee23fff640 1 -- 172.21.15.77:0/2238110139 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fee30036020 con 0x55f4837ea340 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 10 client.15741 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 20 client.15741 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483868120 0x55f48386a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 4 client.15741 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:43:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4837572d0 con 0x55f483868120 2024-03-16T09:43:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee48da7640 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483868120 0x55f48386a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee48da7640 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483868120 0x55f48386a540 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb2675f2c77d44ff server_cookie=cb282a63c7508c23 in_seq=0 out_seq=0 2024-03-16T09:43:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.434+0000 7fee23fff640 10 client.15741 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee23fff640 1 -- 172.21.15.77:0/2238110139 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f4837572d0 con 0x55f483868120 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee23fff640 10 client.15741 handle_command_reply: tid=0 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 client.15741 shutdown 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 msgr2=0x55f4837fba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837fba80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fee300099e0 tx=0x7fee30011d40 comp rx=0 tx=0).stop 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 shutdown_connections 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f483868120 0x55f48386a540 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4837efea0 0x55f4837fa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4837ead30 0x55f4837fa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee23fff640 0 client.15741 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 --2- 172.21.15.77:0/2238110139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f4837ea340 0x55f4837fba80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.437+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 >> 172.21.15.77:0/2238110139 conn(0x55f4837e58b0 msgr2=0x55f4837e93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 shutdown_connections 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4dbf7740 1 -- 172.21.15.77:0/2238110139 wait complete. 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4dbf7740 20 client.15741 trim_cache size 0 max 16384 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:43:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:43:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fee340a2920 msgr2=0x7fee340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fee340a2920 0x7fee340a4de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0b9ef0 tx=0x7fee3c0a8040 comp rx=0 tx=0).stop 2024-03-16T09:43:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 msgr2=0x7fee441a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee441a1150 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee3c02c700 tx=0x7fee3c0328a0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 shutdown_connections 2024-03-16T09:43:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fee340a2920 0x7fee340a4de0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee441022b0 0x7fee441a1150 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee441018f0 0x7fee4419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 --2- 172.21.15.77:0/2397070056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee44100700 0x7fee44199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 >> 172.21.15.77:0/2397070056 conn(0x7fee440fbc70 msgr2=0x7fee440fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 shutdown_connections 2024-03-16T09:43:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:14.438+0000 7fee4b00b640 1 -- 172.21.15.77:0/2397070056 wait complete. 2024-03-16T09:43:14.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:14 smithi077 ceph-mon[32171]: pgmap v386: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:43:14.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:43:14.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:43:14.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:14 smithi129 ceph-mon[35841]: pgmap v386: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:43:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:43:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:43:14.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:14 smithi119 ceph-mon[35632]: pgmap v386: 97 pgs: 97 active+clean; 662 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-16T09:43:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:43:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:43:14.690 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:43:15.010 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-16T09:43:15.078 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-16T09:43:15.118 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T09:43:15.120 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T09:43:15.120 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-16T09:43:15.158 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T09:43:15.162 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:43:15.162 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-16T09:43:15.217 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-03-16T09:43:15.218 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-03-16T09:43:15.219 DEBUG:teuthology.orchestra.run.smithi077:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-03-16T09:43:15.281 INFO:tasks.workunit.client.0.smithi077.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-16T09:43:15.356 INFO:tasks.workunit.client.0.smithi077.stdout: Iozone: Performance Test of File I/O 2024-03-16T09:43:15.356 INFO:tasks.workunit.client.0.smithi077.stdout: Version $Revision: 3.494 $ 2024-03-16T09:43:15.356 INFO:tasks.workunit.client.0.smithi077.stdout: Compiled for 64 bit mode. 2024-03-16T09:43:15.356 INFO:tasks.workunit.client.0.smithi077.stdout: Build: linux-AMD64 2024-03-16T09:43:15.356 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:15.356 INFO:tasks.workunit.client.0.smithi077.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-16T09:43:15.356 INFO:tasks.workunit.client.0.smithi077.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Run began: Sat Mar 16 09:43:15 2024 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Include close in write timing 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Include fsync in write timing 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: File size set to 1048576 kB 2024-03-16T09:43:15.357 INFO:tasks.workunit.client.0.smithi077.stdout: Record Size 16 kB 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: Output is in kBytes/sec 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: Time Resolution = 0.000001 seconds. 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: Processor cache size set to 1024 kBytes. 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: Processor cache line size set to 32 bytes. 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: File stride size set to 17 * record size. 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: Throughput test with 1 process 2024-03-16T09:43:15.358 INFO:tasks.workunit.client.0.smithi077.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-03-16T09:43:17.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:16 smithi119 ceph-mon[35632]: pgmap v387: 97 pgs: 97 active+clean; 663 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T09:43:17.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:16 smithi077 ceph-mon[32171]: pgmap v387: 97 pgs: 97 active+clean; 663 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T09:43:17.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:16 smithi129 ceph-mon[35841]: pgmap v387: 97 pgs: 97 active+clean; 663 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-16T09:43:19.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:18 smithi119 ceph-mon[35632]: pgmap v388: 97 pgs: 97 active+clean; 73 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6.0 MiB/s wr, 7 op/s 2024-03-16T09:43:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:18 smithi077 ceph-mon[32171]: pgmap v388: 97 pgs: 97 active+clean; 73 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6.0 MiB/s wr, 7 op/s 2024-03-16T09:43:19.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:18 smithi129 ceph-mon[35841]: pgmap v388: 97 pgs: 97 active+clean; 73 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6.0 MiB/s wr, 7 op/s 2024-03-16T09:43:21.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:20 smithi119 ceph-mon[35632]: pgmap v389: 97 pgs: 97 active+clean; 173 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 14 MiB/s wr, 9 op/s 2024-03-16T09:43:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:20 smithi077 ceph-mon[32171]: pgmap v389: 97 pgs: 97 active+clean; 173 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 14 MiB/s wr, 9 op/s 2024-03-16T09:43:21.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:20 smithi129 ceph-mon[35841]: pgmap v389: 97 pgs: 97 active+clean; 173 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 14 MiB/s wr, 9 op/s 2024-03-16T09:43:23.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:22 smithi119 ceph-mon[35632]: pgmap v390: 97 pgs: 97 active+clean; 473 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 39 MiB/s wr, 18 op/s 2024-03-16T09:43:23.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:22 smithi077 ceph-mon[32171]: pgmap v390: 97 pgs: 97 active+clean; 473 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 39 MiB/s wr, 18 op/s 2024-03-16T09:43:23.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:22 smithi129 ceph-mon[35841]: pgmap v390: 97 pgs: 97 active+clean; 473 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 39 MiB/s wr, 18 op/s 2024-03-16T09:43:24.238 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:24.238 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 initial writers = 118081.38 kB/sec 2024-03-16T09:43:24.239 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 initial writers = 118078.89 kB/sec 2024-03-16T09:43:24.239 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 118081.38 kB/sec 2024-03-16T09:43:24.239 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 118081.38 kB/sec 2024-03-16T09:43:24.239 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 118081.38 kB/sec 2024-03-16T09:43:24.239 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:43:25.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:24 smithi119 ceph-mon[35632]: pgmap v391: 97 pgs: 97 active+clean; 633 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 53 MiB/s wr, 20 op/s 2024-03-16T09:43:25.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:24 smithi077 ceph-mon[32171]: pgmap v391: 97 pgs: 97 active+clean; 633 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 53 MiB/s wr, 20 op/s 2024-03-16T09:43:25.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:24 smithi129 ceph-mon[35841]: pgmap v391: 97 pgs: 97 active+clean; 633 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 53 MiB/s wr, 20 op/s 2024-03-16T09:43:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:26 smithi119 ceph-mon[35632]: pgmap v392: 97 pgs: 97 active+clean; 805 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 67 MiB/s wr, 25 op/s 2024-03-16T09:43:27.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:26 smithi077 ceph-mon[32171]: pgmap v392: 97 pgs: 97 active+clean; 805 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 67 MiB/s wr, 25 op/s 2024-03-16T09:43:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:26 smithi129 ceph-mon[35841]: pgmap v392: 97 pgs: 97 active+clean; 805 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 67 MiB/s wr, 25 op/s 2024-03-16T09:43:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.533+0000 7fa863430640 1 -- 172.21.15.77:0/1742131249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1029f0 msgr2=0x7fa85c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.533+0000 7fa863430640 1 --2- 172.21.15.77:0/1742131249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1029f0 0x7fa85c104e60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009900 tx=0x7fa84c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.533+0000 7fa863430640 1 -- 172.21.15.77:0/1742131249 shutdown_connections 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.533+0000 7fa863430640 1 --2- 172.21.15.77:0/1742131249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa85c1053a0 0x7fa85c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.533+0000 7fa863430640 1 --2- 172.21.15.77:0/1742131249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1029f0 0x7fa85c104e60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.533+0000 7fa863430640 1 --2- 172.21.15.77:0/1742131249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa85c100080 0x7fa85c1024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.533+0000 7fa863430640 1 -- 172.21.15.77:0/1742131249 >> 172.21.15.77:0/1742131249 conn(0x7fa85c0fbc80 msgr2=0x7fa85c0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 -- 172.21.15.77:0/1742131249 shutdown_connections 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 -- 172.21.15.77:0/1742131249 wait complete. 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 Processor -- start 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 -- start start 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa85c100080 0x7fa85c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa85c1029f0 0x7fa85c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1053a0 0x7fa85c1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c113910 con 0x7fa85c1053a0 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c113790 con 0x7fa85c1029f0 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa863430640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c113a90 con 0x7fa85c100080 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa8619cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1053a0 0x7fa85c1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa8611cc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa85c100080 0x7fa85c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa8619cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1053a0 0x7fa85c1a5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34440/0 (socket says 172.21.15.77:34440) 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa8619cd640 1 -- 172.21.15.77:0/896452882 learned_addr learned my addr 172.21.15.77:0/896452882 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.534+0000 7fa8609cb640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa85c1029f0 0x7fa85c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8619cd640 1 -- 172.21.15.77:0/896452882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa85c100080 msgr2=0x7fa85c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8619cd640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa85c100080 0x7fa85c19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8619cd640 1 -- 172.21.15.77:0/896452882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa85c1029f0 msgr2=0x7fa85c19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8619cd640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa85c1029f0 0x7fa85c19e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8619cd640 1 -- 172.21.15.77:0/896452882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85c1a5b70 con 0x7fa85c1053a0 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8611cc640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa85c100080 0x7fa85c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8619cd640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1053a0 0x7fa85c1a5500 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa85400efd0 tx=0x7fa85400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8527fc640 1 -- 172.21.15.77:0/896452882 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854003b70 con 0x7fa85c1053a0 2024-03-16T09:43:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8527fc640 1 -- 172.21.15.77:0/896452882 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa854003d10 con 0x7fa85c1053a0 2024-03-16T09:43:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85c1a5e60 con 0x7fa85c1053a0 2024-03-16T09:43:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa8527fc640 1 -- 172.21.15.77:0/896452882 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854010980 con 0x7fa85c1053a0 2024-03-16T09:43:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.535+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85c1a6320 con 0x7fa85c1053a0 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.537+0000 7fa8527fc640 1 -- 172.21.15.77:0/896452882 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa854004020 con 0x7fa85c1053a0 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 Processor -- start 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 -- start start 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dcd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dd5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5602d1d32a40 con 0x5602d1dd2ea0 2024-03-16T09:43:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5602d1cdb920 con 0x5602d1dcdd30 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa866016740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5602d1d2eae0 con 0x5602d1dcd340 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8527fc640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa8480a2920 0x7fa8480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8609cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dd5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8609cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dd5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34452/0 (socket says 172.21.15.77:34452) 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8609cb640 1 -- 172.21.15.77:0/3166469066 learned_addr learned my addr 172.21.15.77:0/3166469066 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8611cc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dcd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8619cd640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa8480a2920 0x7fa8480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.538+0000 7fa8527fc640 1 -- 172.21.15.77:0/896452882 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fa8540cee30 con 0x7fa85c1053a0 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8611cc640 1 -- 172.21.15.77:0/3166469066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 msgr2=0x5602d1dcd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dcd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8611cc640 1 -- 172.21.15.77:0/3166469066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 msgr2=0x5602d1dd5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dd5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8611cc640 1 -- 172.21.15.77:0/3166469066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d1cf3460 con 0x5602d1dcdd30 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8619cd640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa8480a2920 0x7fa8480a4de0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa8540a3a50 tx=0x7fa8540665a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd2830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa858009fd0 tx=0x7fa8580118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a10f96d788ce79b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8477fe640 1 -- 172.21.15.77:0/3166469066 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858004030 con 0x5602d1dcdd30 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8477fe640 1 -- 172.21.15.77:0/3166469066 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8580054d0 con 0x5602d1dcdd30 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.539+0000 7fa8477fe640 1 -- 172.21.15.77:0/3166469066 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858004c40 con 0x5602d1dcdd30 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 -- 172.21.15.77:0/3166469066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 msgr2=0x5602d1dd2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd2830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa858009fd0 tx=0x7fa8580118d0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 -- 172.21.15.77:0/3166469066 shutdown_connections 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dd5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd2830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 --2- 172.21.15.77:0/3166469066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dcd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 -- 172.21.15.77:0/3166469066 >> 172.21.15.77:0/3166469066 conn(0x5602d1dc88b0 msgr2=0x5602d1dc8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 -- 172.21.15.77:0/3166469066 shutdown_connections 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 -- 172.21.15.77:0/3166469066 wait complete. 2024-03-16T09:43:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 -- start start 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.540+0000 7fa866016740 1 Processor -- start 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dcd340 0x5602d1dd76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5602d1dd5e50 con 0x5602d1dcd340 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5602d1ceed70 con 0x5602d1dcdd30 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5602d1dd5fd0 con 0x5602d1dd2ea0 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 10 client.0 _ll_register_callbacks cb 0x5602d1a46a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8609cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dcd340 0x5602d1dd76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:56444/0 (socket says 172.21.15.77:56444) 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 -- 172.21.15.77:0/3108070931 learned_addr learned my addr 172.21.15.77:0/3108070931 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 -- 172.21.15.77:0/3108070931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dd2ea0 msgr2=0x5602d1ddebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 -- 172.21.15.77:0/3108070931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dcd340 msgr2=0x5602d1dd76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dcd340 0x5602d1dd76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 -- 172.21.15.77:0/3108070931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d1a47210 con 0x5602d1dcdd30 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dcd340 0x5602d1dd76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd7bf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa858004630 tx=0x7fa85800fc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85067f7bc71f6031 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa844ff9640 1 -- 172.21.15.77:0/3108070931 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858029020 con 0x5602d1dcdd30 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8580054d0 con 0x5602d1dcdd30 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.541+0000 7fa844ff9640 1 -- 172.21.15.77:0/3108070931 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fa858025470 con 0x5602d1dcdd30 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa844ff9640 1 -- 172.21.15.77:0/3108070931 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa858025670 con 0x5602d1dcdd30 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 10 client.25628 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5602d1b13620 con 0x5602d1dcdd30 2024-03-16T09:43:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa844ff9640 1 -- 172.21.15.77:0/3108070931 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8580054d0 con 0x5602d1dcdd30 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa844ff9640 1 -- 172.21.15.77:0/3108070931 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa858004b70 con 0x5602d1dcdd30 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 10 client.25628 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 10 client.25628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 20 client.25628 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1de32f0 0x5602d1de56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 4 client.25628 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602d1d3ff80 con 0x5602d1de32f0 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.542+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1de32f0 0x5602d1de56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.543+0000 7fa844ff9640 10 client.25628 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.543+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1de32f0 0x5602d1de56e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a277950627ef7e7 server_cookie=cbbf4e20ca4a9860 in_seq=0 out_seq=0 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.543+0000 7fa844ff9640 1 -- 172.21.15.77:0/3108070931 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602d1d3ff80 con 0x5602d1de32f0 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.543+0000 7fa844ff9640 10 client.25628 handle_command_reply: tid=0 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 client.25628 shutdown 2024-03-16T09:43:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 msgr2=0x5602d1dd7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd7bf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa858004630 tx=0x7fa85800fc40 comp rx=0 tx=0).stop 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 shutdown_connections 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1de32f0 0x5602d1de56e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1dd7bf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa844ff9640 0 client.25628 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 --2- 172.21.15.77:0/3108070931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dcd340 0x5602d1dd76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 >> 172.21.15.77:0/3108070931 conn(0x5602d1dc88b0 msgr2=0x5602d1dc7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 shutdown_connections 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 1 -- 172.21.15.77:0/3108070931 wait complete. 2024-03-16T09:43:27.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.544+0000 7fa866016740 20 client.25628 trim_cache size 0 max 16384 2024-03-16T09:43:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 Processor -- start 2024-03-16T09:43:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 -- start start 2024-03-16T09:43:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1de9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dea500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1b1990 con 0x5602d1dd2ea0 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1af260 con 0x5602d1dcdd30 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa866016740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1ad0b0 con 0x5602d1dcd340 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8609cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dea500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8609cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dea500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34468/0 (socket says 172.21.15.77:34468) 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8609cb640 1 -- 172.21.15.77:0/1909525399 learned_addr learned my addr 172.21.15.77:0/1909525399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8611cc640 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8609cb640 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1de9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8611cc640 1 -- 172.21.15.77:0/1909525399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 msgr2=0x5602d1de9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8611cc640 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1de9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8611cc640 1 -- 172.21.15.77:0/1909525399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 msgr2=0x5602d1dea500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8611cc640 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dea500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8611cc640 1 -- 172.21.15.77:0/1909525399 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d1dd2300 con 0x5602d1dcdd30 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.559+0000 7fa8609cb640 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dea500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa8611cc640 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeb80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa858000c00 tx=0x7fa858009e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3c04cc35ce10095 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa846ffd640 1 -- 172.21.15.77:0/1909525399 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858029040 con 0x5602d1dcdd30 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa846ffd640 1 -- 172.21.15.77:0/1909525399 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa858005700 con 0x5602d1dcdd30 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa846ffd640 1 -- 172.21.15.77:0/1909525399 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8580195c0 con 0x5602d1dcdd30 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 -- 172.21.15.77:0/1909525399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 msgr2=0x5602d1ddeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeb80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa858000c00 tx=0x7fa858009e20 comp rx=0 tx=0).stop 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 -- 172.21.15.77:0/1909525399 shutdown_connections 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1dea500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeb80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 --2- 172.21.15.77:0/1909525399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1de9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 -- 172.21.15.77:0/1909525399 >> 172.21.15.77:0/1909525399 conn(0x5602d1dc88b0 msgr2=0x5602d1dcc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 -- 172.21.15.77:0/1909525399 shutdown_connections 2024-03-16T09:43:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.560+0000 7fa866016740 1 -- 172.21.15.77:0/1909525399 wait complete. 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 -- start start 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 Processor -- start 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dde570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1ad720 con 0x5602d1dd2ea0 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1aeb90 con 0x5602d1dcdd30 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c1addf0 con 0x5602d1dcd340 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa866016740 10 client.0 _ll_register_callbacks cb 0x7fa858005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34482/0 (socket says 172.21.15.77:34482) 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 -- 172.21.15.77:0/3902756145 learned_addr learned my addr 172.21.15.77:0/3902756145 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dde570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 -- 172.21.15.77:0/3902756145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 msgr2=0x5602d1dde570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dde570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 -- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 msgr2=0x5602d1ddeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.561+0000 7fa8609cb640 1 -- 172.21.15.77:0/3902756145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d1ddd3b0 con 0x5602d1dd2ea0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa8609cb640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddcb80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa84c036040 tx=0x7fa84c045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d062eea0dcbeeb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa844ff9640 1 -- 172.21.15.77:0/3902756145 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c04e4f0 con 0x5602d1dd2ea0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa844ff9640 1 -- 172.21.15.77:0/3902756145 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fa84c03a040 con 0x5602d1dd2ea0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa858005700 con 0x5602d1dd2ea0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa844ff9640 1 -- 172.21.15.77:0/3902756145 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa84c04e690 con 0x5602d1dd2ea0 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 10 client.15759 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5602d1e4a120 con 0x5602d1dd2ea0 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa844ff9640 1 -- 172.21.15.77:0/3902756145 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa858005700 con 0x5602d1dd2ea0 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa844ff9640 1 -- 172.21.15.77:0/3902756145 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa84c0046c0 con 0x5602d1dd2ea0 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 10 client.15759 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 10 client.15759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 20 client.15759 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1e4ab00 0x5602d1e4cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 4 client.15759 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.562+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602d1d3a2d0 con 0x5602d1e4ab00 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.563+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1e4ab00 0x5602d1e4cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.563+0000 7fa844ff9640 10 client.15759 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.563+0000 7fa8611cc640 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1e4ab00 0x5602d1e4cf20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75a68fe43dc9467a server_cookie=c46cc9047252c2c6 in_seq=0 out_seq=0 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.572+0000 7fa844ff9640 1 -- 172.21.15.77:0/3902756145 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602d1d3a2d0 con 0x5602d1e4ab00 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.572+0000 7fa844ff9640 10 client.15759 handle_command_reply: tid=0 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.572+0000 7fa866016740 1 client.15759 shutdown 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 msgr2=0x5602d1ddcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddcb80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa84c036040 tx=0x7fa84c045040 comp rx=0 tx=0).stop 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 shutdown_connections 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5602d1e4ab00 0x5602d1e4cf20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5602d1dd2ea0 0x5602d1ddcb80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5602d1dcdd30 0x5602d1ddeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa844ff9640 0 client.15759 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 --2- 172.21.15.77:0/3902756145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602d1dcd340 0x5602d1dde570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 >> 172.21.15.77:0/3902756145 conn(0x5602d1dc88b0 msgr2=0x5602d1dc7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 shutdown_connections 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 1 -- 172.21.15.77:0/3902756145 wait complete. 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa866016740 20 client.15759 trim_cache size 0 max 16384 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:43:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:43:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa8480a2920 msgr2=0x7fa8480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa8480a2920 0x7fa8480a4de0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa8540a3a50 tx=0x7fa8540665a0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1053a0 msgr2=0x7fa85c1a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1053a0 0x7fa85c1a5500 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa85400efd0 tx=0x7fa85400c6a0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 shutdown_connections 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fa8480a2920 0x7fa8480a4de0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa85c1053a0 0x7fa85c1a5500 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa85c1029f0 0x7fa85c19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 --2- 172.21.15.77:0/896452882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa85c100080 0x7fa85c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.573+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 >> 172.21.15.77:0/896452882 conn(0x7fa85c0fbc80 msgr2=0x7fa85c100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.574+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 shutdown_connections 2024-03-16T09:43:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:27.574+0000 7fa863430640 1 -- 172.21.15.77:0/896452882 wait complete. 2024-03-16T09:43:29.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:28 smithi119 ceph-mon[35632]: pgmap v393: 97 pgs: 97 active+clean; 993 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 83 MiB/s wr, 28 op/s 2024-03-16T09:43:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:28 smithi077 ceph-mon[32171]: pgmap v393: 97 pgs: 97 active+clean; 993 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 83 MiB/s wr, 28 op/s 2024-03-16T09:43:29.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:28 smithi129 ceph-mon[35841]: pgmap v393: 97 pgs: 97 active+clean; 993 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 83 MiB/s wr, 28 op/s 2024-03-16T09:43:30.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:43:30.554 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:43:30.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 -- 172.21.15.77:0/2977065917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 msgr2=0x7f5740106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 --2- 172.21.15.77:0/2977065917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f5740106fe0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5728009900 tx=0x7f572802bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:30.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 -- 172.21.15.77:0/2977065917 shutdown_connections 2024-03-16T09:43:30.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 --2- 172.21.15.77:0/2977065917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5740107520 0x7f5740109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 --2- 172.21.15.77:0/2977065917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f5740106fe0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 --2- 172.21.15.77:0/2977065917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57401022c0 0x7f57401046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 -- 172.21.15.77:0/2977065917 >> 172.21.15.77:0/2977065917 conn(0x7f57400fbc70 msgr2=0x7f57400fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 -- 172.21.15.77:0/2977065917 shutdown_connections 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.675+0000 7f5745377640 1 -- 172.21.15.77:0/2977065917 wait complete. 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 Processor -- start 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 -- start start 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57401022c0 0x7f5740199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f574019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5740107520 0x7f57401a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f574010d2e0 con 0x7f5740104bf0 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f574010d160 con 0x7f5740107520 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f5745377640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f574010d460 con 0x7f57401022c0 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f574019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f574019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34504/0 (socket says 172.21.15.77:34504) 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f57367fc640 1 -- 172.21.15.77:0/2360038378 learned_addr learned my addr 172.21.15.77:0/2360038378 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:30.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.676+0000 7f573effd640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57401022c0 0x7f5740199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f573f7fe640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5740107520 0x7f57401a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f57367fc640 1 -- 172.21.15.77:0/2360038378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57401022c0 msgr2=0x7f5740199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f57367fc640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57401022c0 0x7f5740199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f57367fc640 1 -- 172.21.15.77:0/2360038378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5740107520 msgr2=0x7f57401a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f57367fc640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5740107520 0x7f57401a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f57367fc640 1 -- 172.21.15.77:0/2360038378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57401a1800 con 0x7f5740104bf0 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f573f7fe640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5740107520 0x7f57401a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f57367fc640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f574019a120 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5728036040 tx=0x7f57280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f573cff9640 1 -- 172.21.15.77:0/2360038378 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728043020 con 0x7f5740104bf0 2024-03-16T09:43:30.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f573cff9640 1 -- 172.21.15.77:0/2360038378 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5728004aa0 con 0x7f5740104bf0 2024-03-16T09:43:30.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57401a1a90 con 0x7f5740104bf0 2024-03-16T09:43:30.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f573cff9640 1 -- 172.21.15.77:0/2360038378 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728003ce0 con 0x7f5740104bf0 2024-03-16T09:43:30.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.677+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57401a2030 con 0x7f5740104bf0 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.680+0000 7f573cff9640 1 -- 172.21.15.77:0/2360038378 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5728046020 con 0x7f5740104bf0 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.680+0000 7f5748038740 1 Processor -- start 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.680+0000 7f5748038740 1 -- start start 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.680+0000 7f573cff9640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f572c0a2880 0x7f572c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f573f7fe640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f572c0a2880 0x7f572c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f573cff9640 1 -- 172.21.15.77:0/2360038378 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57280f0e50 con 0x7f5740104bf0 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f5748038740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271dd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f5748038740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f5748038740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271e5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f5748038740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563027142500 con 0x5630271ddb60 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f5748038740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5630270eabe0 con 0x5630271dd170 2024-03-16T09:43:30.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f5748038740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56302713e5a0 con 0x5630271e2cd0 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34508/0 (socket says 172.21.15.77:34508) 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 -- 172.21.15.77:0/3587408282 learned_addr learned my addr 172.21.15.77:0/3587408282 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f573f7fe640 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f573effd640 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271dd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 -- 172.21.15.77:0/3587408282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 msgr2=0x5630271e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 -- 172.21.15.77:0/3587408282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 msgr2=0x5630271dd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271dd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 -- 172.21.15.77:0/3587408282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630270fe4b0 con 0x5630271ddb60 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f573f7fe640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f572c0a2880 0x7f572c0a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5738005fd0 tx=0x7f573800d710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f573effd640 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271dd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57367fc640 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e2660 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f57280b9c00 tx=0x7f572802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21f02629b32c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.681+0000 7f57357fa640 1 -- 172.21.15.77:0/3587408282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57280d6070 con 0x5630271ddb60 2024-03-16T09:43:30.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f57357fa640 1 -- 172.21.15.77:0/3587408282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5728050350 con 0x5630271ddb60 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f57357fa640 1 -- 172.21.15.77:0/3587408282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57280c5f40 con 0x5630271ddb60 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 -- 172.21.15.77:0/3587408282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 msgr2=0x5630271e2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e2660 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f57280b9c00 tx=0x7f572802bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 -- 172.21.15.77:0/3587408282 shutdown_connections 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e2660 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 --2- 172.21.15.77:0/3587408282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271dd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 -- 172.21.15.77:0/3587408282 >> 172.21.15.77:0/3587408282 conn(0x5630271d86e0 msgr2=0x5630271d8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.682+0000 7f5748038740 1 -- 172.21.15.77:0/3587408282 shutdown_connections 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 -- 172.21.15.77:0/3587408282 wait complete. 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 -- start start 2024-03-16T09:43:30.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 Processor -- start 2024-03-16T09:43:30.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271e7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5630271e5a40 con 0x5630271ddb60 2024-03-16T09:43:30.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5630270fe160 con 0x5630271dd170 2024-03-16T09:43:30.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.683+0000 7f5748038740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5630271e5bc0 con 0x5630271e2cd0 2024-03-16T09:43:30.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f5748038740 10 client.0 _ll_register_callbacks cb 0x5630270e2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34520/0 (socket says 172.21.15.77:34520) 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 -- 172.21.15.77:0/3695600417 learned_addr learned my addr 172.21.15.77:0/3695600417 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f573effd640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f573f7fe640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 -- 172.21.15.77:0/3695600417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 msgr2=0x5630271ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271ee9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 -- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 msgr2=0x5630271e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271e7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 -- 172.21.15.77:0/3695600417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563026f207d0 con 0x5630271ddb60 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f573effd640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f573f7fe640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57367fc640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e7970 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f57280c1ab0 tx=0x7f57280bd180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3786dcaa0fea13f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57267fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57267fc640 1 -- 172.21.15.77:0/3695600417 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57280d6070 con 0x5630271ddb60 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f57267fc640 1 -- 172.21.15.77:0/3695600417 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57280a80b0 con 0x5630271ddb60 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.684+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5728050350 con 0x5630271ddb60 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f57267fc640 1 -- 172.21.15.77:0/3695600417 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f57280bdea0 con 0x5630271ddb60 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 10 client.15771 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563026e563f0 con 0x5630271ddb60 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f57267fc640 1 -- 172.21.15.77:0/3695600417 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5728050350 con 0x5630271ddb60 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f57267fc640 1 -- 172.21.15.77:0/3695600417 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f57280d1b80 con 0x5630271ddb60 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 10 client.15771 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 20 client.15771 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5630271f2c30 0x5630271f5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 4 client.15771 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.685+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56302714fa40 con 0x5630271f2c30 2024-03-16T09:43:30.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.686+0000 7f573effd640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5630271f2c30 0x5630271f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.686+0000 7f573effd640 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5630271f2c30 0x5630271f5050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e4e744ea832cf0c server_cookie=717ebe992a2d1fe3 in_seq=0 out_seq=0 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.686+0000 7f57267fc640 10 client.15771 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f57267fc640 1 -- 172.21.15.77:0/3695600417 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56302714fa40 con 0x5630271f2c30 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f57267fc640 10 client.15771 handle_command_reply: tid=0 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 client.15771 shutdown 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 msgr2=0x5630271e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e7970 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f57280c1ab0 tx=0x7f57280bd180 comp rx=0 tx=0).stop 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 shutdown_connections 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5630271f2c30 0x5630271f5050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271e2cd0 0x5630271ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271e7970 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 --2- 172.21.15.77:0/3695600417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271dd170 0x5630271e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 >> 172.21.15.77:0/3695600417 conn(0x5630271d86e0 msgr2=0x5630271d7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:30.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f57267fc640 0 client.15771 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:30.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 shutdown_connections 2024-03-16T09:43:30.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 1 -- 172.21.15.77:0/3695600417 wait complete. 2024-03-16T09:43:30.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.687+0000 7f5748038740 20 client.15771 trim_cache size 0 max 16384 2024-03-16T09:43:30.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 Processor -- start 2024-03-16T09:43:30.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 -- start start 2024-03-16T09:43:30.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271dd170 0x5630271df0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271ddb60 0x5630271ee970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x5630271df600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57401ad6c0 con 0x5630271dd170 2024-03-16T09:43:30.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f57401aaf90 con 0x5630271e2cd0 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f5748038740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57401a8de0 con 0x5630271ddb60 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f573effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271dd170 0x5630271df0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f573effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271dd170 0x5630271df0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34532/0 (socket says 172.21.15.77:34532) 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.708+0000 7f573effd640 1 -- 172.21.15.77:0/1506887122 learned_addr learned my addr 172.21.15.77:0/1506887122 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f573f7fe640 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x5630271df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f573effd640 1 -- 172.21.15.77:0/1506887122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271ddb60 msgr2=0x5630271ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f573effd640 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271ddb60 0x5630271ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f573effd640 1 -- 172.21.15.77:0/1506887122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 msgr2=0x5630271df600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f573effd640 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x5630271df600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f573effd640 1 -- 172.21.15.77:0/1506887122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563026e160f0 con 0x5630271dd170 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f573effd640 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271dd170 0x5630271df0c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x5630271de3c0 tx=0x7f573000a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83db7e19eefbd2aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f5734ff9640 1 -- 172.21.15.77:0/1506887122 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573001b330 con 0x5630271dd170 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f5734ff9640 1 -- 172.21.15.77:0/1506887122 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57300149a0 con 0x5630271dd170 2024-03-16T09:43:30.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.709+0000 7f5734ff9640 1 -- 172.21.15.77:0/1506887122 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730005510 con 0x5630271dd170 2024-03-16T09:43:30.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 -- 172.21.15.77:0/1506887122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271dd170 msgr2=0x5630271df0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271dd170 0x5630271df0c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x5630271de3c0 tx=0x7f573000a2d0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 -- 172.21.15.77:0/1506887122 shutdown_connections 2024-03-16T09:43:30.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x5630271df600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271ddb60 0x5630271ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 --2- 172.21.15.77:0/1506887122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271dd170 0x5630271df0c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 -- 172.21.15.77:0/1506887122 >> 172.21.15.77:0/1506887122 conn(0x5630271d86e0 msgr2=0x5630271d38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 -- 172.21.15.77:0/1506887122 shutdown_connections 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.710+0000 7f5748038740 1 -- 172.21.15.77:0/1506887122 wait complete. 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 -- start start 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 Processor -- start 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271dd170 0x5630271eb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271ebd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x563027256f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57401a9450 con 0x5630271ddb60 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f57401aa8c0 con 0x5630271e2cd0 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57401a9b20 con 0x5630271dd170 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f5748038740 10 client.0 _ll_register_callbacks cb 0x7f57280d4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271ebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271ebd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34546/0 (socket says 172.21.15.77:34546) 2024-03-16T09:43:30.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 -- 172.21.15.77:0/3858436295 learned_addr learned my addr 172.21.15.77:0/3858436295 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f573effd640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271dd170 0x5630271eb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 -- 172.21.15.77:0/3858436295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271dd170 msgr2=0x5630271eb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271dd170 0x5630271eb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 -- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 msgr2=0x563027256f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f573f7fe640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x563027256f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x563027256f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f57367fc640 1 -- 172.21.15.77:0/3858436295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630271099d0 con 0x5630271ddb60 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.711+0000 7f573f7fe640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x563027256f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f57367fc640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271ebd20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5728006010 tx=0x7f57280b9b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b192c2f96e36144 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f57267fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f57267fc640 1 -- 172.21.15.77:0/3858436295 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57280d6040 con 0x5630271ddb60 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f57267fc640 1 -- 172.21.15.77:0/3858436295 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57280a80b0 con 0x5630271ddb60 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57300149a0 con 0x5630271ddb60 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f57267fc640 1 -- 172.21.15.77:0/3858436295 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f57280bd100 con 0x5630271ddb60 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 10 client.15777 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5630272584b0 con 0x5630271ddb60 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f57267fc640 1 -- 172.21.15.77:0/3858436295 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57300149a0 con 0x5630271ddb60 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f57267fc640 1 -- 172.21.15.77:0/3858436295 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f57280c5ab0 con 0x5630271ddb60 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 10 client.15777 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 20 client.15777 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x563027258a30 0x56302725ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 4 client.15777 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.712+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563027149d90 con 0x563027258a30 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f573effd640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x563027258a30 0x56302725ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f57267fc640 10 client.15777 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f573effd640 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x563027258a30 0x56302725ae50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e82637012935298d server_cookie=c36a773577ca2ee7 in_seq=0 out_seq=0 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f57267fc640 1 -- 172.21.15.77:0/3858436295 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563027149d90 con 0x563027258a30 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f57267fc640 10 client.15777 handle_command_reply: tid=0 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 client.15777 shutdown 2024-03-16T09:43:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 msgr2=0x5630271ebd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271ebd20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5728006010 tx=0x7f57280b9b40 comp rx=0 tx=0).stop 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 shutdown_connections 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x563027258a30 0x56302725ae50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5630271e2cd0 0x563027256f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630271ddb60 0x5630271ebd20 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 --2- 172.21.15.77:0/3858436295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5630271dd170 0x5630271eb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f57267fc640 0 client.15777 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 >> 172.21.15.77:0/3858436295 conn(0x5630271d86e0 msgr2=0x5630271e73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 shutdown_connections 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 1 -- 172.21.15.77:0/3858436295 wait complete. 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.713+0000 7f5748038740 20 client.15777 trim_cache size 0 max 16384 2024-03-16T09:43:30.722 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:43:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T09:43:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T09:43:30.723 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:43:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f572c0a2880 msgr2=0x7f572c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f572c0a2880 0x7f572c0a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5738005fd0 tx=0x7f573800d710 comp rx=0 tx=0).stop 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 msgr2=0x7f574019a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f574019a120 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5728036040 tx=0x7f57280042f0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 shutdown_connections 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f572c0a2880 0x7f572c0a4d40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5740107520 0x7f57401a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5740104bf0 0x7f574019a120 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 --2- 172.21.15.77:0/2360038378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57401022c0 0x7f5740199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 >> 172.21.15.77:0/2360038378 conn(0x7f57400fbc70 msgr2=0x7f5740101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 shutdown_connections 2024-03-16T09:43:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.714+0000 7f5745377640 1 -- 172.21.15.77:0/2360038378 wait complete. 2024-03-16T09:43:30.731 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T09:43:30.731 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:43:30.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/441038838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 msgr2=0x7f2ec4109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/441038838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 0x7f2ec4109970 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc009ca0 tx=0x7f2ebc02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/441038838 shutdown_connections 2024-03-16T09:43:30.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/441038838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 0x7f2ec4109970 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/441038838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ec4104c50 0x7f2ec4107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/441038838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 0x7f2ec4104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/441038838 >> 172.21.15.77:0/441038838 conn(0x7f2ec40fbc70 msgr2=0x7f2ec40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/441038838 shutdown_connections 2024-03-16T09:43:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.851+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/441038838 wait complete. 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 Processor -- start 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 -- start start 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 0x7f2ec4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ec4104c50 0x7f2ec4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 0x7f2ec419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410d330 con 0x7f2ec4107580 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410d1b0 con 0x7f2ec4104c50 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec8cb8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410d4b0 con 0x7f2ec4102320 2024-03-16T09:43:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 0x7f2ec4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 0x7f2ec419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 0x7f2ec4197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:50232/0 (socket says 172.21.15.77:50232) 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec259c640 1 -- 172.21.15.77:0/3766431058 learned_addr learned my addr 172.21.15.77:0/3766431058 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.852+0000 7f2ec1d9b640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ec4104c50 0x7f2ec4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec259c640 1 -- 172.21.15.77:0/3766431058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ec4104c50 msgr2=0x7f2ec4197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec259c640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ec4104c50 0x7f2ec4197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec259c640 1 -- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 msgr2=0x7f2ec419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec259c640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 0x7f2ec419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec259c640 1 -- 172.21.15.77:0/3766431058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec419f650 con 0x7f2ec4102320 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec2d9d640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 0x7f2ec419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec259c640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 0x7f2ec4197a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400e9f0 tx=0x7f2eb400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2eb37fe640 1 -- 172.21.15.77:0/3766431058 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb400cd80 con 0x7f2ec4102320 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2eb37fe640 1 -- 172.21.15.77:0/3766431058 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2eb40108e0 con 0x7f2ec4102320 2024-03-16T09:43:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec4100170 con 0x7f2ec4102320 2024-03-16T09:43:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.853+0000 7f2eb37fe640 1 -- 172.21.15.77:0/3766431058 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4018850 con 0x7f2ec4102320 2024-03-16T09:43:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.854+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec4100710 con 0x7f2ec4102320 2024-03-16T09:43:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.855+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec40fd760 con 0x7f2ec4102320 2024-03-16T09:43:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.856+0000 7f2eb37fe640 1 -- 172.21.15.77:0/3766431058 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2eb4010a90 con 0x7f2ec4102320 2024-03-16T09:43:30.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.857+0000 7f2eb37fe640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2eac0a2880 0x7f2eac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:30.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.857+0000 7f2eb37fe640 1 -- 172.21.15.77:0/3766431058 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2eb40cec80 con 0x7f2ec4102320 2024-03-16T09:43:30.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.857+0000 7f2ec1d9b640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2eac0a2880 0x7f2eac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:30.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.858+0000 7f2ec1d9b640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2eac0a2880 0x7f2eac0a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4198f40 tx=0x7f2eb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:30.861+0000 7f2eb37fe640 1 -- 172.21.15.77:0/3766431058 <== mon.1 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2eb409b6e0 con 0x7f2ec4102320 2024-03-16T09:43:30.903 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:30 smithi077 ceph-mon[32171]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 79 MiB/s wr, 27 op/s 2024-03-16T09:43:31.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.147+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ec402d1b0 con 0x7f2ec4102320 2024-03-16T09:43:31.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.148+0000 7f2eb37fe640 1 -- 172.21.15.77:0/3766431058 <== mon.1 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f2eb409f780 con 0x7f2ec4102320 2024-03-16T09:43:31.151 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:43:31.151 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:43:31.152 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:43:31.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2eac0a2880 msgr2=0x7f2eac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2eac0a2880 0x7f2eac0a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4198f40 tx=0x7f2eb8005f90 comp rx=0 tx=0).stop 2024-03-16T09:43:31.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 msgr2=0x7f2ec4197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 0x7f2ec4197a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400e9f0 tx=0x7f2eb400eeb0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 shutdown_connections 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2eac0a2880 0x7f2eac0a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec4107580 0x7f2ec419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ec4104c50 0x7f2ec4197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 --2- 172.21.15.77:0/3766431058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ec4102320 0x7f2ec4197a30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 >> 172.21.15.77:0/3766431058 conn(0x7f2ec40fbc70 msgr2=0x7f2ec4102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.150+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 shutdown_connections 2024-03-16T09:43:31.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.151+0000 7f2ec8cb8640 1 -- 172.21.15.77:0/3766431058 wait complete. 2024-03-16T09:43:31.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T09:43:31.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:30 smithi119 ceph-mon[35632]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 79 MiB/s wr, 27 op/s 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 -- 172.21.15.77:0/3096679611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 msgr2=0x7f54c00fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/3096679611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c00fff00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009930 tx=0x7f54b002bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 -- 172.21.15.77:0/3096679611 shutdown_connections 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/3096679611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54c0103470 0x7f54c0109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/3096679611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54c0102ab0 0x7f54c0102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/3096679611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c00fff00 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 -- 172.21.15.77:0/3096679611 >> 172.21.15.77:0/3096679611 conn(0x7f54c00fb410 msgr2=0x7f54c00fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 -- 172.21.15.77:0/3096679611 shutdown_connections 2024-03-16T09:43:31.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 -- 172.21.15.77:0/3096679611 wait complete. 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.291+0000 7f54c5cc6640 1 Processor -- start 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54c5cc6640 1 -- start start 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54c5cc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54c5cc6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54c0102ab0 0x7f54c0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54c5cc6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54c0103470 0x7f54c019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54c5cc6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f54c010d360 con 0x7f54c00ffae0 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54c5cc6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54c010d1e0 con 0x7f54c0102ab0 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54c5cc6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54c010d4e0 con 0x7f54c0103470 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c0197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34596/0 (socket says 172.21.15.77:34596) 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 -- 172.21.15.77:0/2442078234 learned_addr learned my addr 172.21.15.77:0/2442078234 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54beffd640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54c0102ab0 0x7f54c0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bffff640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54c0103470 0x7f54c019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 -- 172.21.15.77:0/2442078234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54c0103470 msgr2=0x7f54c019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54c0103470 0x7f54c019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 -- 172.21.15.77:0/2442078234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54c0102ab0 msgr2=0x7f54c0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54c0102ab0 0x7f54c0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bf7fe640 1 -- 172.21.15.77:0/2442078234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c019f6d0 con 0x7f54c00ffae0 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54beffd640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54c0102ab0 0x7f54c0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.292+0000 7f54bffff640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54c0103470 0x7f54c019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.293+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c0197b00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f54b0000c00 tx=0x7f54b0005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.293+0000 7f54bcff9640 1 -- 172.21.15.77:0/2442078234 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0043020 con 0x7f54c00ffae0 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.293+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c019f960 con 0x7f54c00ffae0 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.293+0000 7f54bcff9640 1 -- 172.21.15.77:0/2442078234 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54b00043d0 con 0x7f54c00ffae0 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.293+0000 7f54bcff9640 1 -- 172.21.15.77:0/2442078234 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0003b50 con 0x7f54c00ffae0 2024-03-16T09:43:31.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.293+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c019feb0 con 0x7f54c00ffae0 2024-03-16T09:43:31.305 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:30 smithi129 ceph-mon[35841]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 79 MiB/s wr, 27 op/s 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 Processor -- start 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54bcff9640 1 -- 172.21.15.77:0/2442078234 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f54b0046020 con 0x7f54c00ffae0 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 -- start start 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x557680392c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x557680397d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x55768039a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5576802f7490 con 0x5576803983b0 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55768029fc90 con 0x557680392850 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54c898f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5576802f3530 con 0x557680393240 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54bffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x557680392c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54beffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x55768039a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54bffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x557680392c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:56556/0 (socket says 172.21.15.77:56556) 2024-03-16T09:43:31.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54bffff640 1 -- 172.21.15.77:0/3197166569 learned_addr learned my addr 172.21.15.77:0/3197166569 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x557680397d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.296+0000 7f54bcff9640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f54a80a2800 0x7f54a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bf7fe640 1 -- 172.21.15.77:0/3197166569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 msgr2=0x557680392c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x557680392c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bf7fe640 1 -- 172.21.15.77:0/3197166569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 msgr2=0x55768039a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x55768039a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bf7fe640 1 -- 172.21.15.77:0/3197166569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55768035a500 con 0x557680393240 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bffff640 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x557680392c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54beffd640 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x55768039a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bffff640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f54a80a2800 0x7f54a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bcff9640 1 -- 172.21.15.77:0/2442078234 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f54b00f0c30 con 0x7f54c00ffae0 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x557680397d40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f54b00ee710 tx=0x7f54b0088410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2583a33334cb676 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54bffff640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f54a80a2800 0x7f54a80a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x557680393aa0 tx=0x7f54b8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54a5ffb640 1 -- 172.21.15.77:0/3197166569 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b00bd120 con 0x557680393240 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54a5ffb640 1 -- 172.21.15.77:0/3197166569 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54b0088920 con 0x557680393240 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.297+0000 7f54a5ffb640 1 -- 172.21.15.77:0/3197166569 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0088bb0 con 0x557680393240 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 -- 172.21.15.77:0/3197166569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 msgr2=0x557680397d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x557680397d40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f54b00ee710 tx=0x7f54b0088410 comp rx=0 tx=0).stop 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 -- 172.21.15.77:0/3197166569 shutdown_connections 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x55768039a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x557680397d40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 --2- 172.21.15.77:0/3197166569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x557680392c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 -- 172.21.15.77:0/3197166569 >> 172.21.15.77:0/3197166569 conn(0x55768038ddc0 msgr2=0x55768038e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 -- 172.21.15.77:0/3197166569 shutdown_connections 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 -- 172.21.15.77:0/3197166569 wait complete. 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.298+0000 7f54c898f740 1 -- start start 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 1 Processor -- start 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x55768039c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x55768039cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x5576803a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5576802f3530 con 0x5576803983b0 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5576802b3230 con 0x557680392850 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55768039b120 con 0x557680393240 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54c898f740 10 client.0 _ll_register_callbacks cb 0x5576802ef010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x5576803a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x55768039c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54beffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x55768039cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x5576803a3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34620/0 (socket says 172.21.15.77:34620) 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 -- 172.21.15.77:0/2558075385 learned_addr learned my addr 172.21.15.77:0/2558075385 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 -- 172.21.15.77:0/2558075385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 msgr2=0x55768039cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x55768039cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 -- 172.21.15.77:0/2558075385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 msgr2=0x55768039c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x55768039c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 -- 172.21.15.77:0/2558075385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576802c2100 con 0x5576803983b0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x55768039c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f54bffff640 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x5576803a3f00 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f54b8005fd0 tx=0x7f54b8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b092af901a53af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.299+0000 7f549effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f549effd640 1 -- 172.21.15.77:0/2558075385 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b801e420 con 0x5576803983b0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f549effd640 1 -- 172.21.15.77:0/2558075385 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f54b801e5c0 con 0x5576803983b0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54b0088920 con 0x5576803983b0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f549effd640 1 -- 172.21.15.77:0/2558075385 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f54b801e7c0 con 0x5576803983b0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 10 client.15795 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55768000b3f0 con 0x5576803983b0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f549effd640 1 -- 172.21.15.77:0/2558075385 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54b0088920 con 0x5576803983b0 2024-03-16T09:43:31.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f549effd640 1 -- 172.21.15.77:0/2558075385 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f54b801f990 con 0x5576803983b0 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 10 client.15795 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 10 client.15795 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 20 client.15795 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803a88f0 0x5576803aad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 4 client.15795 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576803049d0 con 0x5576803a88f0 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.300+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803a88f0 0x5576803aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.301+0000 7f549effd640 10 client.15795 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.301+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803a88f0 0x5576803aad90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1a52c6aff7cef0d server_cookie=e1ebf2410b2f03aa in_seq=0 out_seq=0 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.301+0000 7f549effd640 1 -- 172.21.15.77:0/2558075385 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5576803049d0 con 0x5576803a88f0 2024-03-16T09:43:31.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.301+0000 7f549effd640 10 client.15795 handle_command_reply: tid=0 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 client.15795 shutdown 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 msgr2=0x5576803a3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x5576803a3f00 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f54b8005fd0 tx=0x7f54b8004ab0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 shutdown_connections 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5576803983b0 0x5576803a3f00 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x55768039cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803a88f0 0x5576803aad90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 --2- 172.21.15.77:0/2558075385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x55768039c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f549effd640 0 client.15795 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 >> 172.21.15.77:0/2558075385 conn(0x55768038ddc0 msgr2=0x55768038cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 shutdown_connections 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 1 -- 172.21.15.77:0/2558075385 wait complete. 2024-03-16T09:43:31.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.302+0000 7f54c898f740 20 client.15795 trim_cache size 0 max 16384 2024-03-16T09:43:31.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 Processor -- start 2024-03-16T09:43:31.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 -- start start 2024-03-16T09:43:31.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x55768038ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x5576803a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x5576803af650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01ab4c0 con 0x55768038ddc0 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01a8d90 con 0x557680392850 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54c898f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01a6be0 con 0x557680393240 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x55768038ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x55768038ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34624/0 (socket says 172.21.15.77:34624) 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 -- 172.21.15.77:0/4232471282 learned_addr learned my addr 172.21.15.77:0/4232471282 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 -- 172.21.15.77:0/4232471282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 msgr2=0x5576803af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/4232471282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x5576803af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 -- 172.21.15.77:0/4232471282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 msgr2=0x5576803a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/4232471282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x5576803a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.322+0000 7f54bf7fe640 1 -- 172.21.15.77:0/4232471282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576800d57d0 con 0x55768038ddc0 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.323+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/4232471282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x55768038ce60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f54b00bd570 tx=0x7f54b00f41c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1417819ad61e1e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.323+0000 7f54a57fa640 1 -- 172.21.15.77:0/4232471282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b00bcf40 con 0x55768038ddc0 2024-03-16T09:43:31.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.323+0000 7f54a57fa640 1 -- 172.21.15.77:0/4232471282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54b00bdce0 con 0x55768038ddc0 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.323+0000 7f54a57fa640 1 -- 172.21.15.77:0/4232471282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b00bd5e0 con 0x55768038ddc0 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 -- 172.21.15.77:0/4232471282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 msgr2=0x55768038ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 --2- 172.21.15.77:0/4232471282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x55768038ce60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f54b00bd570 tx=0x7f54b00f41c0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 -- 172.21.15.77:0/4232471282 shutdown_connections 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 --2- 172.21.15.77:0/4232471282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680393240 0x5576803af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 --2- 172.21.15.77:0/4232471282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680392850 0x5576803a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 --2- 172.21.15.77:0/4232471282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x55768038ce60 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 -- 172.21.15.77:0/4232471282 >> 172.21.15.77:0/4232471282 conn(0x557680394020 msgr2=0x55768038c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 -- 172.21.15.77:0/4232471282 shutdown_connections 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 -- 172.21.15.77:0/4232471282 wait complete. 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 -- start start 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.324+0000 7f54c898f740 1 Processor -- start 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x5576803a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680392850 0x5576803a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680393240 0x5576803a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01a7250 con 0x55768038ddc0 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01a86c0 con 0x557680393240 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54c01a7920 con 0x557680392850 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 10 client.0 _ll_register_callbacks cb 0x7f54b80099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x5576803a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x5576803a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34626/0 (socket says 172.21.15.77:34626) 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 -- 172.21.15.77:0/3803648312 learned_addr learned my addr 172.21.15.77:0/3803648312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54beffd640 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680392850 0x5576803a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bffff640 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680393240 0x5576803a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 -- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680392850 msgr2=0x5576803a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680392850 0x5576803a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 -- 172.21.15.77:0/3803648312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680393240 msgr2=0x5576803a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680393240 0x5576803a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 -- 172.21.15.77:0/3803648312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576803a35a0 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54bf7fe640 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x5576803a1a60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f54b002cdd0 tx=0x7f54b0009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c6e4413c4e9a2c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f549effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f549effd640 1 -- 172.21.15.77:0/3803648312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b00bcf40 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f549effd640 1 -- 172.21.15.77:0/3803648312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f54b00d1680 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.325+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54b00bdce0 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f549effd640 1 -- 172.21.15.77:0/3803648312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f54b00c1730 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 10 client.15801 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55768040f0f0 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f549effd640 1 -- 172.21.15.77:0/3803648312 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54b00bdce0 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f549effd640 1 -- 172.21.15.77:0/3803648312 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f54b00b9c50 con 0x55768038ddc0 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 10 client.15801 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 10 client.15801 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 20 client.15801 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:31.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803983b0 0x5576804116a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 4 client.15801 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:43:31.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.326+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576802fed20 con 0x5576803983b0 2024-03-16T09:43:31.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.327+0000 7f54beffd640 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803983b0 0x5576804116a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54beffd640 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803983b0 0x5576804116a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9a4b8971412d391f server_cookie=d2e261da4d868177 in_seq=0 out_seq=0 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f549effd640 10 client.15801 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f549effd640 1 -- 172.21.15.77:0/3803648312 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576802fed20 con 0x5576803983b0 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f549effd640 10 client.15801 handle_command_reply: tid=0 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 client.15801 shutdown 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 msgr2=0x5576803a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x5576803a1a60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f54b002cdd0 tx=0x7f54b0009680 comp rx=0 tx=0).stop 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 shutdown_connections 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5576803983b0 0x5576804116a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557680393240 0x5576803a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557680392850 0x5576803a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 --2- 172.21.15.77:0/3803648312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55768038ddc0 0x5576803a1a60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 >> 172.21.15.77:0/3803648312 conn(0x557680394020 msgr2=0x5576803918b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f549effd640 0 client.15801 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 shutdown_connections 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.328+0000 7f54c898f740 1 -- 172.21.15.77:0/3803648312 wait complete. 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c898f740 20 client.15801 trim_cache size 0 max 16384 2024-03-16T09:43:31.336 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f54a80a2800 msgr2=0x7f54a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f54a80a2800 0x7f54a80a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x557680393aa0 tx=0x7f54b8004a40 comp rx=0 tx=0).stop 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 msgr2=0x7f54c0197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c0197b00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f54b0000c00 tx=0x7f54b0005e70 comp rx=0 tx=0).stop 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 shutdown_connections 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f54a80a2800 0x7f54a80a4cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54c0103470 0x7f54c019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54c0102ab0 0x7f54c0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 --2- 172.21.15.77:0/2442078234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54c00ffae0 0x7f54c0197b00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 >> 172.21.15.77:0/2442078234 conn(0x7f54c00fb410 msgr2=0x7f54c0105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 shutdown_connections 2024-03-16T09:43:31.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.329+0000 7f54c5cc6640 1 -- 172.21.15.77:0/2442078234 wait complete. 2024-03-16T09:43:31.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T09:43:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 -- 172.21.15.77:0/3033505931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481018d0 msgr2=0x7efe48101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 --2- 172.21.15.77:0/3033505931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481018d0 0x7efe48101d50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efe34009900 tx=0x7efe3402bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 -- 172.21.15.77:0/3033505931 shutdown_connections 2024-03-16T09:43:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 --2- 172.21.15.77:0/3033505931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe48102290 0x7efe48108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 --2- 172.21.15.77:0/3033505931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481018d0 0x7efe48101d50 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 --2- 172.21.15.77:0/3033505931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe481006e0 0x7efe48100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 -- 172.21.15.77:0/3033505931 >> 172.21.15.77:0/3033505931 conn(0x7efe480fbc70 msgr2=0x7efe480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 -- 172.21.15.77:0/3033505931 shutdown_connections 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.469+0000 7efe4ec19640 1 -- 172.21.15.77:0/3033505931 wait complete. 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 Processor -- start 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 -- start start 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481006e0 0x7efe480694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe481018d0 0x7efe4819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe48102290 0x7efe4819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810c1f0 con 0x7efe481006e0 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810c070 con 0x7efe481018d0 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4ec19640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810c370 con 0x7efe48102290 2024-03-16T09:43:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4c9b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481006e0 0x7efe480694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4c9b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481006e0 0x7efe480694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34646/0 (socket says 172.21.15.77:34646) 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe47fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe481018d0 0x7efe4819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4c9b5640 1 -- 172.21.15.77:0/2280929768 learned_addr learned my addr 172.21.15.77:0/2280929768 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.470+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe48102290 0x7efe4819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4c9b5640 1 -- 172.21.15.77:0/2280929768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe48102290 msgr2=0x7efe4819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe48102290 0x7efe4819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4c9b5640 1 -- 172.21.15.77:0/2280929768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe481018d0 msgr2=0x7efe4819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe481018d0 0x7efe4819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4c9b5640 1 -- 172.21.15.77:0/2280929768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe4819f6b0 con 0x7efe481006e0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe47fff640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe481018d0 0x7efe4819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481006e0 0x7efe480694b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00d950 tx=0x7efe3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe45ffb640 1 -- 172.21.15.77:0/2280929768 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c017020 con 0x7efe481006e0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe45ffb640 1 -- 172.21.15.77:0/2280929768 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3c009c30 con 0x7efe481006e0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe45ffb640 1 -- 172.21.15.77:0/2280929768 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c00f6b0 con 0x7efe481006e0 2024-03-16T09:43:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe4819f940 con 0x7efe481006e0 2024-03-16T09:43:31.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.471+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe4819fe60 con 0x7efe481006e0 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe45ffb640 1 -- 172.21.15.77:0/2280929768 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efe3c01a020 con 0x7efe481006e0 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 Processor -- start 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 -- start start 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b234850 0x56010b234c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b235240 0x56010b239d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b23c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56010b199490 con 0x56010b23a3b0 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56010b141c90 con 0x56010b234850 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe51809740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56010b195530 con 0x56010b235240 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b23c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe4d1b6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b234850 0x56010b234c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b23c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34648/0 (socket says 172.21.15.77:34648) 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 -- 172.21.15.77:0/1184614053 learned_addr learned my addr 172.21.15.77:0/1184614053 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe45ffb640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b235240 0x56010b239d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 -- 172.21.15.77:0/1184614053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b235240 msgr2=0x56010b239d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b235240 0x56010b239d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 -- 172.21.15.77:0/1184614053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b234850 msgr2=0x56010b234c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b234850 0x56010b234c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe47fff640 1 -- 172.21.15.77:0/1184614053 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010b1fc500 con 0x56010b23a3b0 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.474+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b234850 0x56010b234c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.475+0000 7efe45ffb640 1 -- 172.21.15.77:0/2280929768 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7efe3c0ce910 con 0x7efe481006e0 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.475+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.475+0000 7efe47fff640 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b23c840 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efe34009900 tx=0x7efe34005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bc3f33fec5ac27 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.475+0000 7efe32ffd640 1 -- 172.21.15.77:0/1184614053 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34045020 con 0x56010b23a3b0 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.475+0000 7efe32ffd640 1 -- 172.21.15.77:0/1184614053 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe34005040 con 0x56010b23a3b0 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.475+0000 7efe32ffd640 1 -- 172.21.15.77:0/1184614053 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34004a10 con 0x56010b23a3b0 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.475+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efe380a2800 0x7efe380a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efe40009fd0 tx=0x7efe40004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 -- 172.21.15.77:0/1184614053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 msgr2=0x56010b23c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b23c840 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efe34009900 tx=0x7efe34005d50 comp rx=0 tx=0).stop 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 -- 172.21.15.77:0/1184614053 shutdown_connections 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b23c840 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b235240 0x56010b239d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 --2- 172.21.15.77:0/1184614053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b234850 0x56010b234c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 -- 172.21.15.77:0/1184614053 >> 172.21.15.77:0/1184614053 conn(0x56010b22fdc0 msgr2=0x56010b2301c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 -- 172.21.15.77:0/1184614053 shutdown_connections 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 -- 172.21.15.77:0/1184614053 wait complete. 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 -- start start 2024-03-16T09:43:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.476+0000 7efe51809740 1 Processor -- start 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe51809740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b234850 0x56010b23ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe51809740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b23f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe51809740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b23a3b0 0x56010b2461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe51809740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56010b23d300 con 0x56010b234850 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe51809740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56010b155230 con 0x56010b235240 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe51809740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56010b23d480 con 0x56010b23a3b0 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe51809740 10 client.0 _ll_register_callbacks cb 0x56010b191010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe4c9b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b234850 0x56010b23ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe47fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b23f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe4c9b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b234850 0x56010b23ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34664/0 (socket says 172.21.15.77:34664) 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe4c9b5640 1 -- 172.21.15.77:0/1321464076 learned_addr learned my addr 172.21.15.77:0/1321464076 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b23a3b0 0x56010b2461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe47fff640 1 -- 172.21.15.77:0/1321464076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b23a3b0 msgr2=0x56010b2461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe47fff640 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b23a3b0 0x56010b2461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe47fff640 1 -- 172.21.15.77:0/1321464076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b234850 msgr2=0x56010b23ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe47fff640 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b234850 0x56010b23ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe47fff640 1 -- 172.21.15.77:0/1321464076 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56010b164100 con 0x56010b235240 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b234850 0x56010b23ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe47fff640 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b23f1f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe34036040 tx=0x7efe340049e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b93812bf3bd03509 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe27fff640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.477+0000 7efe27fff640 1 -- 172.21.15.77:0/1321464076 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34045040 con 0x56010b235240 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe34005040 con 0x56010b235240 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe27fff640 1 -- 172.21.15.77:0/1321464076 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7efe34041480 con 0x56010b235240 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe27fff640 1 -- 172.21.15.77:0/1321464076 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efe3405b050 con 0x56010b235240 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 10 client.25673 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56010aead3f0 con 0x56010b235240 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe27fff640 1 -- 172.21.15.77:0/1321464076 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe34005040 con 0x56010b235240 2024-03-16T09:43:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe27fff640 1 -- 172.21.15.77:0/1321464076 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7efe3404a4e0 con 0x56010b235240 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 10 client.25673 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 10 client.25673 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 20 client.25673 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b24a700 0x56010b24caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 4 client.25673 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56010b1a69d0 con 0x56010b24a700 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.478+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b24a700 0x56010b24caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.479+0000 7efe27fff640 10 client.25673 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:43:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.479+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b24a700 0x56010b24caf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b223da84d731fd5d server_cookie=d66e4bffecc96471 in_seq=0 out_seq=0 2024-03-16T09:43:31.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.479+0000 7efe27fff640 1 -- 172.21.15.77:0/1321464076 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56010b1a69d0 con 0x56010b24a700 2024-03-16T09:43:31.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.479+0000 7efe27fff640 10 client.25673 handle_command_reply: tid=0 2024-03-16T09:43:31.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.479+0000 7efe51809740 1 client.25673 shutdown 2024-03-16T09:43:31.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 msgr2=0x56010b23f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b23f1f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe34036040 tx=0x7efe340049e0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 shutdown_connections 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b24a700 0x56010b24caf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b23a3b0 0x56010b2461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe27fff640 0 client.25673 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b23f1f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 --2- 172.21.15.77:0/1321464076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b234850 0x56010b23ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 >> 172.21.15.77:0/1321464076 conn(0x56010b22fdc0 msgr2=0x56010b22efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 shutdown_connections 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 1 -- 172.21.15.77:0/1321464076 wait complete. 2024-03-16T09:43:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.480+0000 7efe51809740 20 client.25673 trim_cache size 0 max 16384 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 Processor -- start 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 -- start start 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b233730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b235240 0x56010b2461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b23a3b0 0x56010b251360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe481ab400 con 0x56010b235240 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a8cd0 con 0x56010b23a3b0 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe51809740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a6b20 con 0x56010b234850 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b235240 0x56010b2461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b235240 0x56010b2461b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34678/0 (socket says 172.21.15.77:34678) 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 -- 172.21.15.77:0/198154100 learned_addr learned my addr 172.21.15.77:0/198154100 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b23a3b0 0x56010b251360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b233730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 -- 172.21.15.77:0/198154100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 msgr2=0x56010b233730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b233730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 -- 172.21.15.77:0/198154100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b23a3b0 msgr2=0x56010b251360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b23a3b0 0x56010b251360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.498+0000 7efe47fff640 1 -- 172.21.15.77:0/198154100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010b22b640 con 0x56010b235240 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.499+0000 7efe47fff640 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b235240 0x56010b2461b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7efe3404a930 tx=0x7efe34005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eec530b7ecf9741 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.499+0000 7efe327fc640 1 -- 172.21.15.77:0/198154100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34045040 con 0x56010b235240 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.499+0000 7efe327fc640 1 -- 172.21.15.77:0/198154100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3404ab20 con 0x56010b235240 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.499+0000 7efe327fc640 1 -- 172.21.15.77:0/198154100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34005040 con 0x56010b235240 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- 172.21.15.77:0/198154100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b235240 msgr2=0x56010b2461b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b235240 0x56010b2461b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7efe3404a930 tx=0x7efe34005d50 comp rx=0 tx=0).stop 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- 172.21.15.77:0/198154100 shutdown_connections 2024-03-16T09:43:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b23a3b0 0x56010b251360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b235240 0x56010b2461b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 --2- 172.21.15.77:0/198154100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b233730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- 172.21.15.77:0/198154100 >> 172.21.15.77:0/198154100 conn(0x56010b22fdc0 msgr2=0x56010b233020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- 172.21.15.77:0/198154100 shutdown_connections 2024-03-16T09:43:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- 172.21.15.77:0/198154100 wait complete. 2024-03-16T09:43:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- start start 2024-03-16T09:43:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 Processor -- start 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b243350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b243890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b2448a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a7190 con 0x56010b23a3b0 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a8600 con 0x56010b235240 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe481a7860 con 0x56010b234850 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.500+0000 7efe51809740 10 client.0 _ll_register_callbacks cb 0x7efe40009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b2448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe47fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b243890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b2448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34680/0 (socket says 172.21.15.77:34680) 2024-03-16T09:43:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4c9b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b243350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe47fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b243890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:56666/0 (socket says 172.21.15.77:56666) 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 -- 172.21.15.77:0/4138635393 learned_addr learned my addr 172.21.15.77:0/4138635393 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 -- 172.21.15.77:0/4138635393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 msgr2=0x56010b243350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b243350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 -- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 msgr2=0x56010b243890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b243890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 -- 172.21.15.77:0/4138635393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56010af777d0 con 0x56010b23a3b0 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe47fff640 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b243890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe4d1b6640 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b2448a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7efe4000fc60 tx=0x7efe40004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484abc58a3fd0bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe27fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe27fff640 1 -- 172.21.15.77:0/4138635393 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4002d020 con 0x56010b23a3b0 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe27fff640 1 -- 172.21.15.77:0/4138635393 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7efe4001d9b0 con 0x56010b23a3b0 2024-03-16T09:43:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe3404ab20 con 0x56010b23a3b0 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe27fff640 1 -- 172.21.15.77:0/4138635393 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efe4001dbb0 con 0x56010b23a3b0 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe51809740 10 client.15819 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.501+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56010b246030 con 0x56010b23a3b0 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe27fff640 1 -- 172.21.15.77:0/4138635393 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3404ab20 con 0x56010b23a3b0 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe27fff640 1 -- 172.21.15.77:0/4138635393 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7efe40034420 con 0x56010b23a3b0 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe51809740 10 client.15819 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe51809740 10 client.15819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe51809740 20 client.15819 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe51809740 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b2b0db0 0x56010b2b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe51809740 4 client.15819 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56010b1a0d20 con 0x56010b2b0db0 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b2b0db0 0x56010b2b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe27fff640 10 client.15819 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:43:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.502+0000 7efe4c9b5640 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b2b0db0 0x56010b2b31d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=62c55dc00c08070f server_cookie=782b387eda691538 in_seq=0 out_seq=0 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe27fff640 1 -- 172.21.15.77:0/4138635393 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56010b1a0d20 con 0x56010b2b0db0 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe27fff640 10 client.15819 handle_command_reply: tid=0 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 client.15819 shutdown 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 msgr2=0x56010b2448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b2448a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7efe4000fc60 tx=0x7efe40004ab0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 shutdown_connections 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56010b2b0db0 0x56010b2b31d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56010b23a3b0 0x56010b2448a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe27fff640 0 client.15819 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56010b235240 0x56010b243890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 --2- 172.21.15.77:0/4138635393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56010b234850 0x56010b243350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 >> 172.21.15.77:0/4138635393 conn(0x56010b22fdc0 msgr2=0x56010b2338b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 shutdown_connections 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 1 -- 172.21.15.77:0/4138635393 wait complete. 2024-03-16T09:43:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.503+0000 7efe51809740 20 client.15819 trim_cache size 0 max 16384 2024-03-16T09:43:31.513 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efe380a2800 msgr2=0x7efe380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efe380a2800 0x7efe380a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efe40009fd0 tx=0x7efe40004a40 comp rx=0 tx=0).stop 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481006e0 msgr2=0x7efe480694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481006e0 0x7efe480694b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00d950 tx=0x7efe3c00de10 comp rx=0 tx=0).stop 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 shutdown_connections 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe48102290 0x7efe4819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe481018d0 0x7efe4819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 --2- 172.21.15.77:0/2280929768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe481006e0 0x7efe480694b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 >> 172.21.15.77:0/2280929768 conn(0x7efe480fbc70 msgr2=0x7efe48104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 shutdown_connections 2024-03-16T09:43:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.504+0000 7efe4ec19640 1 -- 172.21.15.77:0/2280929768 wait complete. 2024-03-16T09:43:31.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T09:43:31.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 -- 172.21.15.77:0/1597094668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 msgr2=0x7fb298101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1597094668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298101d50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009900 tx=0x7fb28c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:31.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 -- 172.21.15.77:0/1597094668 shutdown_connections 2024-03-16T09:43:31.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1597094668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb298102290 0x7fb298108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1597094668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298101d50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1597094668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2981006e0 0x7fb298100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 -- 172.21.15.77:0/1597094668 >> 172.21.15.77:0/1597094668 conn(0x7fb2980fbc70 msgr2=0x7fb2980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 -- 172.21.15.77:0/1597094668 shutdown_connections 2024-03-16T09:43:31.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 -- 172.21.15.77:0/1597094668 wait complete. 2024-03-16T09:43:31.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 Processor -- start 2024-03-16T09:43:31.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.633+0000 7fb29dadc640 1 -- start start 2024-03-16T09:43:31.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb29dadc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2981006e0 0x7fb298197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb29dadc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb29dadc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb298102290 0x7fb29819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb29dadc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810c210 con 0x7fb2981018d0 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb29dadc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810c090 con 0x7fb2981006e0 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb29dadc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810c390 con 0x7fb298102290 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34702/0 (socket says 172.21.15.77:34702) 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 -- 172.21.15.77:0/1197788787 learned_addr learned my addr 172.21.15.77:0/1197788787 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb2977fe640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2981006e0 0x7fb298197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb297fff640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb298102290 0x7fb29819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 -- 172.21.15.77:0/1197788787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb298102290 msgr2=0x7fb29819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb298102290 0x7fb29819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 -- 172.21.15.77:0/1197788787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2981006e0 msgr2=0x7fb298197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2981006e0 0x7fb298197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 -- 172.21.15.77:0/1197788787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29819f660 con 0x7fb2981018d0 2024-03-16T09:43:31.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb296ffd640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298197f80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006010 tx=0x7fb28c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.634+0000 7fb294ff9640 1 -- 172.21.15.77:0/1197788787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c043020 con 0x7fb2981018d0 2024-03-16T09:43:31.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.635+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29819f8f0 con 0x7fb2981018d0 2024-03-16T09:43:31.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.635+0000 7fb294ff9640 1 -- 172.21.15.77:0/1197788787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb28c0043d0 con 0x7fb2981018d0 2024-03-16T09:43:31.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.635+0000 7fb294ff9640 1 -- 172.21.15.77:0/1197788787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c003b80 con 0x7fb2981018d0 2024-03-16T09:43:31.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.635+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29819fe40 con 0x7fb2981018d0 2024-03-16T09:43:31.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.637+0000 7fb294ff9640 1 -- 172.21.15.77:0/1197788787 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb28c046020 con 0x7fb2981018d0 2024-03-16T09:43:31.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.637+0000 7fb2a07a5740 1 Processor -- start 2024-03-16T09:43:31.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.637+0000 7fb294ff9640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb2800a2800 0x7fb2800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.637+0000 7fb2a07a5740 1 -- start start 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb297fff640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb2800a2800 0x7fb2800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccd240 0x55cf8bcd1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcd4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cf8bc31490 con 0x55cf8bccd240 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55cf8bbd9c90 con 0x55cf8bcd23b0 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cf8bc2d530 con 0x55cf8bccc850 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb294ff9640 1 -- 172.21.15.77:0/1197788787 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb28c0f0c60 con 0x7fb2981018d0 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb296ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccd240 0x55cf8bcd1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:50364/0 (socket says 172.21.15.77:50364) 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb296ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccd240 0x55cf8bcd1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34704/0 (socket says 172.21.15.77:34704) 2024-03-16T09:43:31.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 -- 172.21.15.77:0/694712321 learned_addr learned my addr 172.21.15.77:0/694712321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb297fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcd4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 -- 172.21.15.77:0/694712321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bcd23b0 msgr2=0x55cf8bcd4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcd4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 -- 172.21.15.77:0/694712321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccd240 msgr2=0x55cf8bcd1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccd240 0x55cf8bcd1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 -- 172.21.15.77:0/694712321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf8bc94500 con 0x55cf8bccc850 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb296ffd640 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccd240 0x55cf8bcd1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb297fff640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb2800a2800 0x7fb2800a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb290005fd0 tx=0x7fb290011bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.638+0000 7fb2977fe640 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcccc70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb288009c20 tx=0x7fb288011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba024b2af123876b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb27dffb640 1 -- 172.21.15.77:0/694712321 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb288004220 con 0x55cf8bccc850 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb27dffb640 1 -- 172.21.15.77:0/694712321 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2880043c0 con 0x55cf8bccc850 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb27dffb640 1 -- 172.21.15.77:0/694712321 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2880197a0 con 0x55cf8bccc850 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 -- 172.21.15.77:0/694712321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 msgr2=0x55cf8bcccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcccc70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb288009c20 tx=0x7fb288011810 comp rx=0 tx=0).stop 2024-03-16T09:43:31.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 -- 172.21.15.77:0/694712321 shutdown_connections 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcd4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccd240 0x55cf8bcd1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/694712321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcccc70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 -- 172.21.15.77:0/694712321 >> 172.21.15.77:0/694712321 conn(0x55cf8bcc7dc0 msgr2=0x55cf8bcc81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 -- 172.21.15.77:0/694712321 shutdown_connections 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.639+0000 7fb2a07a5740 1 -- 172.21.15.77:0/694712321 wait complete. 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 -- start start 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 Processor -- start 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccc850 0x55cf8bcd6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcd7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcde270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cf8bcd5360 con 0x55cf8bccc850 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55cf8bbed230 con 0x55cf8bccd240 2024-03-16T09:43:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cf8bcd54e0 con 0x55cf8bcd23b0 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2a07a5740 10 client.0 _ll_register_callbacks cb 0x55cf8bc29010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccc850 0x55cf8bcd6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccc850 0x55cf8bcd6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34720/0 (socket says 172.21.15.77:34720) 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb2977fe640 1 -- 172.21.15.77:0/1354748026 learned_addr learned my addr 172.21.15.77:0/1354748026 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb296ffd640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcd7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.640+0000 7fb297fff640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcde270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2977fe640 1 -- 172.21.15.77:0/1354748026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bcd23b0 msgr2=0x55cf8bcde270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2977fe640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcde270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2977fe640 1 -- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 msgr2=0x55cf8bcd7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2977fe640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcd7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2977fe640 1 -- 172.21.15.77:0/1354748026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf8bbfc100 con 0x55cf8bccc850 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb296ffd640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcd7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2977fe640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccc850 0x55cf8bcd6d30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb28801b040 tx=0x7fb288003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a5392fce087ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb276ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb276ffd640 1 -- 172.21.15.77:0/1354748026 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb288028070 con 0x55cf8bccc850 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb276ffd640 1 -- 172.21.15.77:0/1354748026 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb288005040 con 0x55cf8bccc850 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2880043c0 con 0x55cf8bccc850 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb276ffd640 1 -- 172.21.15.77:0/1354748026 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb288034050 con 0x55cf8bccc850 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2a07a5740 10 client.15831 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cf8b9453f0 con 0x55cf8bccc850 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.641+0000 7fb276ffd640 1 -- 172.21.15.77:0/1354748026 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2880043c0 con 0x55cf8bccc850 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb276ffd640 1 -- 172.21.15.77:0/1354748026 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb2880049f0 con 0x55cf8bccc850 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb2a07a5740 10 client.15831 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb2a07a5740 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb2a07a5740 20 client.15831 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bce4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb2a07a5740 4 client.15831 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf8bc3e9d0 con 0x55cf8bce28b0 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb296ffd640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bce4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb276ffd640 10 client.15831 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.642+0000 7fb296ffd640 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bce4ce0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c239d3eb3c2bdaa5 server_cookie=2bd4221b38c65d0f in_seq=0 out_seq=0 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb276ffd640 1 -- 172.21.15.77:0/1354748026 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf8bc3e9d0 con 0x55cf8bce28b0 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb276ffd640 10 client.15831 handle_command_reply: tid=0 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 client.15831 shutdown 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccc850 msgr2=0x55cf8bcd6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccc850 0x55cf8bcd6d30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb28801b040 tx=0x7fb288003820 comp rx=0 tx=0).stop 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 shutdown_connections 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bce4ce0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb276ffd640 0 client.15831 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcde270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcd7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/1354748026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bccc850 0x55cf8bcd6d30 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 >> 172.21.15.77:0/1354748026 conn(0x55cf8bcc7dc0 msgr2=0x55cf8bcc6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 shutdown_connections 2024-03-16T09:43:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 1 -- 172.21.15.77:0/1354748026 wait complete. 2024-03-16T09:43:31.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.643+0000 7fb2a07a5740 20 client.15831 trim_cache size 0 max 16384 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 Processor -- start 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 -- start start 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcd71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcde230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcc2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981ab460 con 0x55cf8bcd23b0 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a8d30 con 0x55cf8bccd240 2024-03-16T09:43:31.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a6b80 con 0x55cf8bccc850 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcc2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcc2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34732/0 (socket says 172.21.15.77:34732) 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 -- 172.21.15.77:0/215480114 learned_addr learned my addr 172.21.15.77:0/215480114 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb296ffd640 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcde230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2977fe640 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcd71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 -- 172.21.15.77:0/215480114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 msgr2=0x55cf8bcd71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcd71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 -- 172.21.15.77:0/215480114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 msgr2=0x55cf8bcde230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcde230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb297fff640 1 -- 172.21.15.77:0/215480114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf8bcc3640 con 0x55cf8bcd23b0 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.662+0000 7fb2977fe640 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcd71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.663+0000 7fb297fff640 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcc2e70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb29000fcb0 tx=0x7fb29001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ce3845206aec48 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.663+0000 7fb27d7fa640 1 -- 172.21.15.77:0/215480114 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29002d070 con 0x55cf8bcd23b0 2024-03-16T09:43:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.663+0000 7fb27d7fa640 1 -- 172.21.15.77:0/215480114 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb29000fce0 con 0x55cf8bcd23b0 2024-03-16T09:43:31.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.663+0000 7fb27d7fa640 1 -- 172.21.15.77:0/215480114 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29001a040 con 0x55cf8bcd23b0 2024-03-16T09:43:31.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 -- 172.21.15.77:0/215480114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 msgr2=0x55cf8bcc2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcc2e70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb29000fcb0 tx=0x7fb29001f000 comp rx=0 tx=0).stop 2024-03-16T09:43:31.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 -- 172.21.15.77:0/215480114 shutdown_connections 2024-03-16T09:43:31.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bcc2e70 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcde230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/215480114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcd71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 -- 172.21.15.77:0/215480114 >> 172.21.15.77:0/215480114 conn(0x55cf8bcc7dc0 msgr2=0x55cf8bcd6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 -- 172.21.15.77:0/215480114 shutdown_connections 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 -- 172.21.15.77:0/215480114 wait complete. 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 -- start start 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.664+0000 7fb2a07a5740 1 Processor -- start 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcdb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcdb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bd46c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a71f0 con 0x55cf8bcd23b0 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a8660 con 0x55cf8bccd240 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a78c0 con 0x55cf8bccc850 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 10 client.0 _ll_register_callbacks cb 0x7fb28c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bd46c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb296ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcdb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bd46c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34744/0 (socket says 172.21.15.77:34744) 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb296ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcdb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:56712/0 (socket says 172.21.15.77:56712) 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 -- 172.21.15.77:0/3423728816 learned_addr learned my addr 172.21.15.77:0/3423728816 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2977fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcdb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 -- 172.21.15.77:0/3423728816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 msgr2=0x55cf8bcdb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcdb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 -- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 msgr2=0x55cf8bcdb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcdb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 -- 172.21.15.77:0/3423728816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf8ba0f7d0 con 0x55cf8bcd23b0 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb296ffd640 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcdb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb297fff640 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bd46c10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb29002b040 tx=0x7fb290004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a6967149271177 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb276ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb276ffd640 1 -- 172.21.15.77:0/3423728816 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29001a070 con 0x55cf8bcd23b0 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb276ffd640 1 -- 172.21.15.77:0/3423728816 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb290028600 con 0x55cf8bcd23b0 2024-03-16T09:43:31.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.665+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb29000fce0 con 0x55cf8bcd23b0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb276ffd640 1 -- 172.21.15.77:0/3423728816 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb29002d070 con 0x55cf8bcd23b0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 10 client.15837 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cf8bd48200 con 0x55cf8bcd23b0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb276ffd640 1 -- 172.21.15.77:0/3423728816 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb29000fce0 con 0x55cf8bcd23b0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb276ffd640 1 -- 172.21.15.77:0/3423728816 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb290027070 con 0x55cf8bcd23b0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 10 client.15837 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 20 client.15837 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bd4ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 4 client.15837 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf8bc38d20 con 0x55cf8bce28b0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.666+0000 7fb2977fe640 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bd4ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2977fe640 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bd4ac20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe483e4df8c7c383 server_cookie=b0ddb7c415232d44 in_seq=0 out_seq=0 2024-03-16T09:43:31.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb276ffd640 10 client.15837 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb276ffd640 1 -- 172.21.15.77:0/3423728816 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf8bc38d20 con 0x55cf8bce28b0 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb276ffd640 10 client.15837 handle_command_reply: tid=0 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 client.15837 shutdown 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 msgr2=0x55cf8bd46c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bd46c10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb29002b040 tx=0x7fb290004ee0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 shutdown_connections 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55cf8bce28b0 0x55cf8bd4ac20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb276ffd640 0 client.15837 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cf8bcd23b0 0x55cf8bd46c10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf8bccd240 0x55cf8bcdb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 --2- 172.21.15.77:0/3423728816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf8bccc850 0x55cf8bcdb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 >> 172.21.15.77:0/3423728816 conn(0x55cf8bcc7dc0 msgr2=0x55cf8bccb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 shutdown_connections 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 1 -- 172.21.15.77:0/3423728816 wait complete. 2024-03-16T09:43:31.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.667+0000 7fb2a07a5740 20 client.15837 trim_cache size 0 max 16384 2024-03-16T09:43:31.677 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:43:31.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb2800a2800 msgr2=0x7fb2800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb2800a2800 0x7fb2800a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb290005fd0 tx=0x7fb290011bc0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 msgr2=0x7fb298197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298197f80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006010 tx=0x7fb28c005e40 comp rx=0 tx=0).stop 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 shutdown_connections 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb2800a2800 0x7fb2800a4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb298102290 0x7fb29819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2981018d0 0x7fb298197f80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 --2- 172.21.15.77:0/1197788787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2981006e0 0x7fb298197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 >> 172.21.15.77:0/1197788787 conn(0x7fb2980fbc70 msgr2=0x7fb298068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 shutdown_connections 2024-03-16T09:43:31.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:31.668+0000 7fb29dadc640 1 -- 172.21.15.77:0/1197788787 wait complete. 2024-03-16T09:43:32.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:31 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3766431058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:43:32.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:31 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3766431058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:43:32.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:31 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3766431058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:43:32.683 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T09:43:32.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:43:32.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T09:43:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 -- 172.21.15.77:0/1346011821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec01006e0 msgr2=0x7f6ec0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/1346011821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec01006e0 0x7f6ec0100b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009900 tx=0x7f6eb002bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 -- 172.21.15.77:0/1346011821 shutdown_connections 2024-03-16T09:43:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/1346011821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec0102290 0x7f6ec0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/1346011821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec01018d0 0x7f6ec0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/1346011821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec01006e0 0x7f6ec0100b00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 -- 172.21.15.77:0/1346011821 >> 172.21.15.77:0/1346011821 conn(0x7f6ec00fbc70 msgr2=0x7f6ec00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 -- 172.21.15.77:0/1346011821 shutdown_connections 2024-03-16T09:43:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.800+0000 7f6ec64ea640 1 -- 172.21.15.77:0/1346011821 wait complete. 2024-03-16T09:43:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 Processor -- start 2024-03-16T09:43:32.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 -- start start 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec01006e0 0x7f6ec0196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec01018d0 0x7f6ec0196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec0102290 0x7f6ec019df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec010c210 con 0x7f6ec0102290 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec010c090 con 0x7f6ec01018d0 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.801+0000 7f6ec64ea640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec010c390 con 0x7f6ec01006e0 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec0102290 0x7f6ec019df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec01006e0 0x7f6ec0196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec0102290 0x7f6ec019df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34770/0 (socket says 172.21.15.77:34770) 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec01006e0 0x7f6ec0196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:50404/0 (socket says 172.21.15.77:50404) 2024-03-16T09:43:32.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 -- 172.21.15.77:0/373270774 learned_addr learned my addr 172.21.15.77:0/373270774 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec01018d0 0x7f6ec0196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 -- 172.21.15.77:0/373270774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec01006e0 msgr2=0x7f6ec0196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec01006e0 0x7f6ec0196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 -- 172.21.15.77:0/373270774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec01018d0 msgr2=0x7f6ec0196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec01018d0 0x7f6ec0196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 -- 172.21.15.77:0/373270774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec019e5b0 con 0x7f6ec0102290 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ebffff640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec01006e0 0x7f6ec0196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec4a87640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec0102290 0x7f6ec019df40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800eb10 tx=0x7f6eb800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ebd7fa640 1 -- 172.21.15.77:0/373270774 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8004b20 con 0x7f6ec0102290 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ebd7fa640 1 -- 172.21.15.77:0/373270774 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eb8004cc0 con 0x7f6ec0102290 2024-03-16T09:43:32.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec019e8a0 con 0x7f6ec0102290 2024-03-16T09:43:32.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ebd7fa640 1 -- 172.21.15.77:0/373270774 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8010690 con 0x7f6ec0102290 2024-03-16T09:43:32.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.802+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec019edf0 con 0x7f6ec0102290 2024-03-16T09:43:32.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.805+0000 7f6ebd7fa640 1 -- 172.21.15.77:0/373270774 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6eb8005020 con 0x7f6ec0102290 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.805+0000 7f6ec91b4740 1 Processor -- start 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.805+0000 7f6ec91b4740 1 -- start start 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84536db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d8453be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d8453e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555d8449b0d0 con 0x555d84537380 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555d84443c70 con 0x555d84536990 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555d84497170 con 0x555d8453c4f0 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d8453be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebd7fa640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea80a2800 0x7f6ea80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d8453be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34772/0 (socket says 172.21.15.77:34772) 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 -- 172.21.15.77:0/4108290390 learned_addr learned my addr 172.21.15.77:0/4108290390 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84536db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d8453e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ec4a87640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea80a2800 0x7f6ea80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebd7fa640 1 -- 172.21.15.77:0/373270774 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6eb80cf110 con 0x7f6ec0102290 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 -- 172.21.15.77:0/4108290390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 msgr2=0x555d8453e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d8453e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 -- 172.21.15.77:0/4108290390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 msgr2=0x555d84536db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84536db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 -- 172.21.15.77:0/4108290390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d842777e0 con 0x555d84537380 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84536db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.806+0000 7f6ebffff640 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d8453be80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009900 tx=0x7f6eb00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7903a57fd8f42a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ea67fc640 1 -- 172.21.15.77:0/4108290390 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0045020 con 0x555d84537380 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec4a87640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea80a2800 0x7f6ea80a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8097f20 tx=0x7f6eb8009a70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ea67fc640 1 -- 172.21.15.77:0/4108290390 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eb002cc90 con 0x555d84537380 2024-03-16T09:43:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ea67fc640 1 -- 172.21.15.77:0/4108290390 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0004920 con 0x555d84537380 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec91b4740 1 -- 172.21.15.77:0/4108290390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 msgr2=0x555d8453be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d8453be80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009900 tx=0x7f6eb00095f0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec91b4740 1 -- 172.21.15.77:0/4108290390 shutdown_connections 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d8453e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d8453be80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/4108290390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84536db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.807+0000 7f6ec91b4740 1 -- 172.21.15.77:0/4108290390 >> 172.21.15.77:0/4108290390 conn(0x555d844fec50 msgr2=0x555d84532300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 -- 172.21.15.77:0/4108290390 shutdown_connections 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 -- 172.21.15.77:0/4108290390 wait complete. 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 -- start start 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 Processor -- start 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84540e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84537380 0x555d845413b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d8453c4f0 0x555d845483f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555d8453f440 con 0x555d8453c4f0 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555d844570d0 con 0x555d84536990 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555d8453f5c0 con 0x555d84537380 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ec91b4740 10 client.0 _ll_register_callbacks cb 0x555d84420760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ebf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d8453c4f0 0x555d845483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ebf7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84540e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.808+0000 7f6ebf7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84540e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:56748/0 (socket says 172.21.15.77:56748) 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebf7fe640 1 -- 172.21.15.77:0/1799360693 learned_addr learned my addr 172.21.15.77:0/1799360693 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebffff640 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84537380 0x555d845413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebf7fe640 1 -- 172.21.15.77:0/1799360693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84537380 msgr2=0x555d845413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84537380 0x555d845413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebf7fe640 1 -- 172.21.15.77:0/1799360693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d8453c4f0 msgr2=0x555d845483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d8453c4f0 0x555d845483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebf7fe640 1 -- 172.21.15.77:0/1799360693 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d842d7f70 con 0x555d84536990 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84540e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400f430 tx=0x7f6eb40096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c7562464b3ae7e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6e9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:43:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/1799360693 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb401a040 con 0x555d84536990 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb002cc90 con 0x555d84536990 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/1799360693 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6eb4004260 con 0x555d84536990 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/1799360693 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6eb400fe70 con 0x555d84536990 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ec91b4740 10 client.25709 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.809+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555d8416f170 con 0x555d84536990 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/1799360693 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eb002cc90 con 0x555d84536990 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/1799360693 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6eb4021430 con 0x555d84536990 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ec91b4740 10 client.25709 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ec91b4740 10 client.25709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ec91b4740 20 client.25709 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d8454ead0 0x555d84550f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ec91b4740 4 client.25709 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d844a8610 con 0x555d8454ead0 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ebffff640 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d8454ead0 0x555d84550f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6e9f7fe640 10 client.25709 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.810+0000 7f6ebffff640 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d8454ead0 0x555d84550f70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a066cfb77a05619e server_cookie=463e136491df5733 in_seq=0 out_seq=0 2024-03-16T09:43:32.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/1799360693 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d844a8610 con 0x555d8454ead0 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6e9f7fe640 10 client.25709 handle_command_reply: tid=0 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 client.25709 shutdown 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 msgr2=0x555d84540e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84540e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400f430 tx=0x7f6eb40096c0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 shutdown_connections 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d8454ead0 0x555d84550f70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d8453c4f0 0x555d845483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6e9f7fe640 0 client.25709 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84537380 0x555d845413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/1799360693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84540e70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.811+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 >> 172.21.15.77:0/1799360693 conn(0x555d844fec50 msgr2=0x555d845314e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.812+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 shutdown_connections 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.812+0000 7f6ec91b4740 1 -- 172.21.15.77:0/1799360693 wait complete. 2024-03-16T09:43:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.812+0000 7f6ec91b4740 20 client.25709 trim_cache size 0 max 16384 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 Processor -- start 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 -- start start 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84536990 0x555d84535c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d845483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d8453c4f0 0x555d84540a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01a9310 con 0x555d84537380 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01a6be0 con 0x555d8453c4f0 2024-03-16T09:43:32.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01a4a30 con 0x555d84536990 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d845483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d845483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34792/0 (socket says 172.21.15.77:34792) 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebffff640 1 -- 172.21.15.77:0/3582287540 learned_addr learned my addr 172.21.15.77:0/3582287540 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d8453c4f0 0x555d84540a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84536990 0x555d84535c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebffff640 1 -- 172.21.15.77:0/3582287540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84536990 msgr2=0x555d84535c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebffff640 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84536990 0x555d84535c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebffff640 1 -- 172.21.15.77:0/3582287540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d8453c4f0 msgr2=0x555d84540a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.836+0000 7f6ebffff640 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d8453c4f0 0x555d84540a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.837+0000 7f6ebffff640 1 -- 172.21.15.77:0/3582287540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d84536190 con 0x555d84537380 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.837+0000 7f6ebffff640 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d845483b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0005ab0 tx=0x7f6eb0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547ac68ebf19ef97 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.837+0000 7f6ea5ffb640 1 -- 172.21.15.77:0/3582287540 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0045040 con 0x555d84537380 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.837+0000 7f6ea5ffb640 1 -- 172.21.15.77:0/3582287540 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eb002cc90 con 0x555d84537380 2024-03-16T09:43:32.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.837+0000 7f6ea5ffb640 1 -- 172.21.15.77:0/3582287540 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0041770 con 0x555d84537380 2024-03-16T09:43:32.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 -- 172.21.15.77:0/3582287540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 msgr2=0x555d845483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d845483b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0005ab0 tx=0x7f6eb0000bc0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 -- 172.21.15.77:0/3582287540 shutdown_connections 2024-03-16T09:43:32.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d8453c4f0 0x555d84540a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d845483b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/3582287540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d84536990 0x555d84535c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 -- 172.21.15.77:0/3582287540 >> 172.21.15.77:0/3582287540 conn(0x555d844fec50 msgr2=0x555d84535540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 -- 172.21.15.77:0/3582287540 shutdown_connections 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 -- 172.21.15.77:0/3582287540 wait complete. 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 -- start start 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.838+0000 7f6ec91b4740 1 Processor -- start 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84547bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d84548100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ec91b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d84546240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01a50a0 con 0x555d84537380 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01a6510 con 0x555d84536990 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ec91b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec01a5770 con 0x555d8453c4f0 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ec91b4740 10 client.0 _ll_register_callbacks cb 0x7f6eb4009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebf7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84547bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d84548100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebf7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84547bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:56770/0 (socket says 172.21.15.77:56770) 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebf7fe640 1 -- 172.21.15.77:0/85250270 learned_addr learned my addr 172.21.15.77:0/85250270 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d84546240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebffff640 1 -- 172.21.15.77:0/85250270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 msgr2=0x555d84546240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebffff640 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d84546240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebffff640 1 -- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 msgr2=0x555d84547bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebffff640 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84547bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebffff640 1 -- 172.21.15.77:0/85250270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d84546880 con 0x555d84537380 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84547bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6ebffff640 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d84548100 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x555d84523fe0 tx=0x7f6eb0009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e7fcd94d74d604 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.839+0000 7f6e9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/85250270 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0045040 con 0x555d84537380 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/85250270 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6eb0003800 con 0x555d84537380 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb002cc90 con 0x555d84537380 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/85250270 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6eb0003a40 con 0x555d84537380 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 10 client.15855 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555d845b57d0 con 0x555d84537380 2024-03-16T09:43:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/85250270 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eb002cc90 con 0x555d84537380 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/85250270 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f6eb0053070 con 0x555d84537380 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 10 client.15855 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 20 client.15855 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d845b5d00 0x555d845b8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 4 client.15855 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d844a2960 con 0x555d845b5d00 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.840+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d845b5d00 0x555d845b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.841+0000 7f6e9f7fe640 10 client.15855 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.841+0000 7f6ebf7fe640 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d845b5d00 0x555d845b8120 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f71f5caf7b8268da server_cookie=83002a05f38f4801 in_seq=0 out_seq=0 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6e9f7fe640 1 -- 172.21.15.77:0/85250270 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d844a2960 con 0x555d845b5d00 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6e9f7fe640 10 client.15855 handle_command_reply: tid=0 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 client.15855 shutdown 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 msgr2=0x555d84548100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d84548100 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x555d84523fe0 tx=0x7f6eb0009680 comp rx=0 tx=0).stop 2024-03-16T09:43:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 shutdown_connections 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555d845b5d00 0x555d845b8120 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555d8453c4f0 0x555d84546240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555d84537380 0x555d84548100 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 --2- 172.21.15.77:0/85250270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d84536990 0x555d84547bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 >> 172.21.15.77:0/85250270 conn(0x555d844fec50 msgr2=0x555d845409f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6e9f7fe640 0 client.15855 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 shutdown_connections 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 1 -- 172.21.15.77:0/85250270 wait complete. 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.842+0000 7f6ec91b4740 20 client.15855 trim_cache size 0 max 16384 2024-03-16T09:43:32.850 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:43:32.851 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T09:43:32.851 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "7b6951b1-aa9e-435e-874b-af5827c61da0", 2024-03-16T09:43:32.851 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T09:43:32.851 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:43:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea80a2800 msgr2=0x7f6ea80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea80a2800 0x7f6ea80a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8097f20 tx=0x7f6eb8009a70 comp rx=0 tx=0).stop 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec0102290 msgr2=0x7f6ec019df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec0102290 0x7f6ec019df40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800eb10 tx=0x7f6eb800c2d0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 shutdown_connections 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f6ea80a2800 0x7f6ea80a4cc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec0102290 0x7f6ec019df40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec01018d0 0x7f6ec0196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 --2- 172.21.15.77:0/373270774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ec01006e0 0x7f6ec0196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 >> 172.21.15.77:0/373270774 conn(0x7f6ec00fbc70 msgr2=0x7f6ec0104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 shutdown_connections 2024-03-16T09:43:32.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.843+0000 7f6ec64ea640 1 -- 172.21.15.77:0/373270774 wait complete. 2024-03-16T09:43:32.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:43:32.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/2545577696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 msgr2=0x7f8df4108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/2545577696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4108b80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009ca0 tx=0x7f8dec02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/2545577696 shutdown_connections 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/2545577696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4108b80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/2545577696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8df4101930 0x7f8df4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/2545577696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8df4100c20 0x7f8df4101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/2545577696 >> 172.21.15.77:0/2545577696 conn(0x7f8df40fb3d0 msgr2=0x7f8df40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/2545577696 shutdown_connections 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.981+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/2545577696 wait complete. 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 Processor -- start 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 -- start start 2024-03-16T09:43:32.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8df4100c20 0x7f8df41046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8df4101930 0x7f8df4102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8df410c330 con 0x7f8df41022f0 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8df410c1b0 con 0x7f8df4100c20 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfc0f1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8df410c4b0 con 0x7f8df4101930 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfa68e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfa68e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4103250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34838/0 (socket says 172.21.15.77:34838) 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8dfa68e640 1 -- 172.21.15.77:0/1342045021 learned_addr learned my addr 172.21.15.77:0/1342045021 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8df968c640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8df4101930 0x7f8df4102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.982+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8df4100c20 0x7f8df41046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfa68e640 1 -- 172.21.15.77:0/1342045021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8df4101930 msgr2=0x7f8df4102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8df4101930 0x7f8df4102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfa68e640 1 -- 172.21.15.77:0/1342045021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8df4100c20 msgr2=0x7f8df41046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8df4100c20 0x7f8df41046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfa68e640 1 -- 172.21.15.77:0/1342045021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df4103ae0 con 0x7f8df41022f0 2024-03-16T09:43:32.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4103250 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8dec032380 tx=0x7f8dec032960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8deaffd640 1 -- 172.21.15.77:0/1342045021 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec041020 con 0x7f8df41022f0 2024-03-16T09:43:32.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8deaffd640 1 -- 172.21.15.77:0/1342045021 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8dec003e20 con 0x7f8df41022f0 2024-03-16T09:43:32.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df41a1c50 con 0x7f8df41022f0 2024-03-16T09:43:32.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8deaffd640 1 -- 172.21.15.77:0/1342045021 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec039040 con 0x7f8df41022f0 2024-03-16T09:43:32.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.983+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df41a21a0 con 0x7f8df41022f0 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.985+0000 7f8deaffd640 1 -- 172.21.15.77:0/1342045021 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8dec044020 con 0x7f8df41022f0 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.986+0000 7f8deaffd640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8de00a2800 0x7f8de00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.986+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8de00a2800 0x7f8de00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.986+0000 7f8deaffd640 1 -- 172.21.15.77:0/1342045021 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8dec0f0db0 con 0x7f8df41022f0 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.986+0000 7f8dfedba740 1 Processor -- start 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.986+0000 7f8dfedba740 1 -- start start 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.986+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8de00a2800 0x7f8de00a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8df4069920 tx=0x7f8df0009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b88c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b891660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b894160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfedba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c10b7f1500 con 0x55c10b88cb60 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfedba740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c10b799be0 con 0x55c10b891cd0 2024-03-16T09:43:32.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfedba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c10b7ed5a0 con 0x55c10b88c170 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b891660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b891660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34844/0 (socket says 172.21.15.77:34844) 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 -- 172.21.15.77:0/574553039 learned_addr learned my addr 172.21.15.77:0/574553039 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b894160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b88c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 -- 172.21.15.77:0/574553039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 msgr2=0x55c10b88c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b88c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 -- 172.21.15.77:0/574553039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 msgr2=0x55c10b894160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b894160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 -- 172.21.15.77:0/574553039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10b7ad4b0 con 0x55c10b88cb60 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b894160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8df968c640 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b891660 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009a40 tx=0x7f8de4009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31c926ec5349a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8ddffff640 1 -- 172.21.15.77:0/574553039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4017070 con 0x55c10b88cb60 2024-03-16T09:43:32.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8ddffff640 1 -- 172.21.15.77:0/574553039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8de4013470 con 0x55c10b88cb60 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.987+0000 7f8ddffff640 1 -- 172.21.15.77:0/574553039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4012630 con 0x55c10b88cb60 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 -- 172.21.15.77:0/574553039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 msgr2=0x55c10b891660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b891660 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009a40 tx=0x7f8de4009f00 comp rx=0 tx=0).stop 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 -- 172.21.15.77:0/574553039 shutdown_connections 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b894160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b891660 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 --2- 172.21.15.77:0/574553039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b88c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 -- 172.21.15.77:0/574553039 >> 172.21.15.77:0/574553039 conn(0x55c10b8876e0 msgr2=0x55c10b887ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 -- 172.21.15.77:0/574553039 shutdown_connections 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.988+0000 7f8dfedba740 1 -- 172.21.15.77:0/574553039 wait complete. 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 -- start start 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 Processor -- start 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b896690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b896bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b89dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c10b894c80 con 0x55c10b88cb60 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c10b7ad160 con 0x55c10b891cd0 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c10b894e00 con 0x55c10b88c170 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfedba740 10 client.0 _ll_register_callbacks cb 0x55c10b791150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8df968c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b896bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8df968c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b896bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34846/0 (socket says 172.21.15.77:34846) 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8df968c640 1 -- 172.21.15.77:0/754373622 learned_addr learned my addr 172.21.15.77:0/754373622 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.989+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b89dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b896690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df968c640 1 -- 172.21.15.77:0/754373622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 msgr2=0x55c10b896690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df968c640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b896690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df968c640 1 -- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 msgr2=0x55c10b89dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df968c640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b89dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df968c640 1 -- 172.21.15.77:0/754373622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10b5cf7d0 con 0x55c10b88cb60 2024-03-16T09:43:32.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b89dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b896690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8df968c640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b896bd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009330 tx=0x7f8de4009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537b9c2d62a7c9e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8ddd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/754373622 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4017050 con 0x55c10b88cb60 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/754373622 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8de401c450 con 0x55c10b88cb60 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de4013470 con 0x55c10b88cb60 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/754373622 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8de401c650 con 0x55c10b88cb60 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8dfedba740 10 client.15867 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c10b5053f0 con 0x55c10b88cb60 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/754373622 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8de401cca0 con 0x55c10b88cb60 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.990+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/754373622 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8de4030ad0 con 0x55c10b88cb60 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8dfedba740 10 client.15867 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8dfedba740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:33.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8dfedba740 20 client.15867 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8dfedba740 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b8a1e70 0x55c10b8a4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8dfedba740 4 client.15867 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8de4013470 con 0x55c10b8a1e70 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b8a1e70 0x55c10b8a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8ddd7fa640 10 client.15867 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.991+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b8a1e70 0x55c10b8a4260 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdbe5f3443c2327f server_cookie=b6ccc845a1b03d49 in_seq=0 out_seq=0 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/754373622 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f8de4013470 con 0x55c10b8a1e70 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8ddd7fa640 10 client.15867 handle_command_reply: tid=0 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 client.15867 shutdown 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 msgr2=0x55c10b896bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b896bd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009330 tx=0x7f8de4009740 comp rx=0 tx=0).stop 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 shutdown_connections 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b8a1e70 0x55c10b8a4260 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b891cd0 0x55c10b89dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8ddd7fa640 0 client.15867 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:33.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b896bd0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 --2- 172.21.15.77:0/754373622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b896690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.992+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 >> 172.21.15.77:0/754373622 conn(0x55c10b8876e0 msgr2=0x55c10b886900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:33.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.993+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 shutdown_connections 2024-03-16T09:43:33.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.993+0000 7f8dfedba740 1 -- 172.21.15.77:0/754373622 wait complete. 2024-03-16T09:43:33.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:32.993+0000 7f8dfedba740 20 client.15867 trim_cache size 0 max 16384 2024-03-16T09:43:33.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:32 smithi077 ceph-mon[32171]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 75 MiB/s wr, 29 op/s 2024-03-16T09:43:33.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 Processor -- start 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 -- start start 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b88e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88cb60 0x55c10b88e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b891cd0 0x55c10b89dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41ad770 con 0x55c10b891cd0 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41ab040 con 0x55c10b88cb60 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.012+0000 7f8dfedba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41a8e90 con 0x55c10b88c170 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b891cd0 0x55c10b89dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b891cd0 0x55c10b89dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34854/0 (socket says 172.21.15.77:34854) 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 -- 172.21.15.77:0/2226478334 learned_addr learned my addr 172.21.15.77:0/2226478334 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 -- 172.21.15.77:0/2226478334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 msgr2=0x55c10b88e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/2226478334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b88e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 -- 172.21.15.77:0/2226478334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88cb60 msgr2=0x55c10b88e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/2226478334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88cb60 0x55c10b88e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 -- 172.21.15.77:0/2226478334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10b8a45d0 con 0x55c10b891cd0 2024-03-16T09:43:33.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/2226478334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b891cd0 0x55c10b89dbd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0bd460 tx=0x7f8dec02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3e4f5c62a309ad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:33.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8ddf7fe640 1 -- 172.21.15.77:0/2226478334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0d6070 con 0x55c10b891cd0 2024-03-16T09:43:33.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8ddf7fe640 1 -- 172.21.15.77:0/2226478334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8dec0bd760 con 0x55c10b891cd0 2024-03-16T09:43:33.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.013+0000 7f8ddf7fe640 1 -- 172.21.15.77:0/2226478334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec004700 con 0x55c10b891cd0 2024-03-16T09:43:33.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 -- 172.21.15.77:0/2226478334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b891cd0 msgr2=0x55c10b89dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 --2- 172.21.15.77:0/2226478334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b891cd0 0x55c10b89dbd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0bd460 tx=0x7f8dec02e040 comp rx=0 tx=0).stop 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 -- 172.21.15.77:0/2226478334 shutdown_connections 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 --2- 172.21.15.77:0/2226478334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b891cd0 0x55c10b89dbd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 --2- 172.21.15.77:0/2226478334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88cb60 0x55c10b88e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 --2- 172.21.15.77:0/2226478334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b88c170 0x55c10b88e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 -- 172.21.15.77:0/2226478334 >> 172.21.15.77:0/2226478334 conn(0x55c10b8876e0 msgr2=0x55c10b896690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 -- 172.21.15.77:0/2226478334 shutdown_connections 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.014+0000 7f8dfedba740 1 -- 172.21.15.77:0/2226478334 wait complete. 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 -- start start 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 Processor -- start 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88c170 0x55c10b89ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b89bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b891cd0 0x55c10b89cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41a9500 con 0x55c10b88cb60 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41aa970 con 0x55c10b88c170 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41a9bd0 con 0x55c10b891cd0 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfedba740 10 client.0 _ll_register_callbacks cb 0x7f8dec0b92e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b89bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b89bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34866/0 (socket says 172.21.15.77:34866) 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 -- 172.21.15.77:0/3507041703 learned_addr learned my addr 172.21.15.77:0/3507041703 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88c170 0x55c10b89ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8dfa68e640 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b891cd0 0x55c10b89cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 -- 172.21.15.77:0/3507041703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b891cd0 msgr2=0x55c10b89cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b891cd0 0x55c10b89cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 -- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88c170 msgr2=0x55c10b89ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88c170 0x55c10b89ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.015+0000 7f8df968c640 1 -- 172.21.15.77:0/3507041703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10b4c50f0 con 0x55c10b88cb60 2024-03-16T09:43:33.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8df968c640 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b89bf70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8de4030e90 tx=0x7f8de40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f6fabbb9b62dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8ddd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/3507041703 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4017050 con 0x55c10b88cb60 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/3507041703 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8de40124f0 con 0x55c10b88cb60 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dec0bd760 con 0x55c10b88cb60 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/3507041703 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8de400ee50 con 0x55c10b88cb60 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 10 client.15873 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c10b7b89d0 con 0x55c10b88cb60 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/3507041703 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8dec0bd760 con 0x55c10b88cb60 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/3507041703 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8de401cab0 con 0x55c10b88cb60 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 10 client.15873 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 20 client.15873 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b908eb0 0x55c10b90b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:33.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 4 client.15873 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.016+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c10b7fea40 con 0x55c10b908eb0 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b908eb0 0x55c10b90b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8ddd7fa640 10 client.15873 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8df9e8d640 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b908eb0 0x55c10b90b2d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd9563e97f52d1c6 server_cookie=90d18e20985016fc in_seq=0 out_seq=0 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8ddd7fa640 1 -- 172.21.15.77:0/3507041703 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c10b7fea40 con 0x55c10b908eb0 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8ddd7fa640 10 client.15873 handle_command_reply: tid=0 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 client.15873 shutdown 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 msgr2=0x55c10b89bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b89bf70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8de4030e90 tx=0x7f8de40047d0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 shutdown_connections 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55c10b908eb0 0x55c10b90b2d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b891cd0 0x55c10b89cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c10b88cb60 0x55c10b89bf70 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8ddd7fa640 0 client.15873 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 --2- 172.21.15.77:0/3507041703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c10b88c170 0x55c10b89ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 >> 172.21.15.77:0/3507041703 conn(0x55c10b8876e0 msgr2=0x55c10b896630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 shutdown_connections 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.017+0000 7f8dfedba740 1 -- 172.21.15.77:0/3507041703 wait complete. 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfedba740 20 client.15873 trim_cache size 0 max 16384 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stdout: "7b6951b1-aa9e-435e-874b-af5827c61da0": { 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "7b6951b1-aa9e-435e-874b-af5827c61da0", 2024-03-16T09:43:33.027 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T09:43:33.028 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:43:33.028 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:43:33.028 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:43:33.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8de00a2800 msgr2=0x7f8de00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8de00a2800 0x7f8de00a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8df4069920 tx=0x7f8df0009110 comp rx=0 tx=0).stop 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 msgr2=0x7f8df4103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4103250 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8dec032380 tx=0x7f8dec032960 comp rx=0 tx=0).stop 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 shutdown_connections 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8de00a2800 0x7f8de00a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df41022f0 0x7f8df4103250 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8df4101930 0x7f8df4102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 --2- 172.21.15.77:0/1342045021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8df4100c20 0x7f8df41046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 >> 172.21.15.77:0/1342045021 conn(0x7f8df40fb3d0 msgr2=0x7f8df40ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 shutdown_connections 2024-03-16T09:43:33.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:43:33.018+0000 7f8dfc0f1640 1 -- 172.21.15.77:0/1342045021 wait complete. 2024-03-16T09:43:33.032 INFO:tasks.cephfs.filesystem:scrub status for tag:7b6951b1-aa9e-435e-874b-af5827c61da0 - {'path': '/', 'tag': '7b6951b1-aa9e-435e-874b-af5827c61da0', 'options': 'recursive,force'} 2024-03-16T09:43:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:32 smithi119 ceph-mon[35632]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 75 MiB/s wr, 29 op/s 2024-03-16T09:43:33.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:32 smithi129 ceph-mon[35841]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 75 MiB/s wr, 29 op/s 2024-03-16T09:43:34.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:33 smithi119 ceph-mon[35632]: scrub queued for path: / 2024-03-16T09:43:34.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:33 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:34.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:33 smithi119 ceph-mon[35632]: scrub summary: active paths [/] 2024-03-16T09:43:34.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:33 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:33 smithi077 ceph-mon[32171]: scrub queued for path: / 2024-03-16T09:43:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:33 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:33 smithi077 ceph-mon[32171]: scrub summary: active paths [/] 2024-03-16T09:43:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:33 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:33 smithi129 ceph-mon[35841]: scrub queued for path: / 2024-03-16T09:43:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:33 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:33 smithi129 ceph-mon[35841]: scrub summary: active paths [/] 2024-03-16T09:43:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:33 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:43:35.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:34 smithi119 ceph-mon[35632]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 57 MiB/s wr, 23 op/s 2024-03-16T09:43:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:34 smithi077 ceph-mon[32171]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 57 MiB/s wr, 23 op/s 2024-03-16T09:43:35.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:34 smithi129 ceph-mon[35841]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 57 MiB/s wr, 23 op/s 2024-03-16T09:43:36.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:35 smithi119 ceph-mon[35632]: scrub summary: idle 2024-03-16T09:43:36.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:35 smithi077 ceph-mon[32171]: scrub summary: idle 2024-03-16T09:43:36.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:35 smithi129 ceph-mon[35841]: scrub summary: idle 2024-03-16T09:43:37.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:36 smithi119 ceph-mon[35632]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 63 MiB/s wr, 26 op/s 2024-03-16T09:43:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:36 smithi077 ceph-mon[32171]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 63 MiB/s wr, 26 op/s 2024-03-16T09:43:37.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:36 smithi129 ceph-mon[35841]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 63 MiB/s wr, 26 op/s 2024-03-16T09:43:37.994 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:37.994 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 rewriters = 131754.09 kB/sec 2024-03-16T09:43:37.994 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 rewriters = 131750.93 kB/sec 2024-03-16T09:43:37.994 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 131754.09 kB/sec 2024-03-16T09:43:37.994 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 131754.09 kB/sec 2024-03-16T09:43:37.994 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 131754.09 kB/sec 2024-03-16T09:43:37.994 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:43:39.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:38 smithi119 ceph-mon[35632]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 78 MiB/s wr, 29 op/s 2024-03-16T09:43:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:38 smithi077 ceph-mon[32171]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 78 MiB/s wr, 29 op/s 2024-03-16T09:43:39.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:38 smithi129 ceph-mon[35841]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 78 MiB/s wr, 29 op/s 2024-03-16T09:43:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 -- 172.21.15.77:0/4037195721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 msgr2=0x7f3f80101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 --2- 172.21.15.77:0/4037195721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80101d70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009900 tx=0x7f3f7002bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 -- 172.21.15.77:0/4037195721 shutdown_connections 2024-03-16T09:43:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 --2- 172.21.15.77:0/4037195721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f801022b0 0x7f3f80108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 --2- 172.21.15.77:0/4037195721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80101d70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 --2- 172.21.15.77:0/4037195721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80100700 0x7f3f80100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 -- 172.21.15.77:0/4037195721 >> 172.21.15.77:0/4037195721 conn(0x7f3f800fbc70 msgr2=0x7f3f800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 -- 172.21.15.77:0/4037195721 shutdown_connections 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 -- 172.21.15.77:0/4037195721 wait complete. 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 Processor -- start 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 -- start start 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80100700 0x7f3f80196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.669+0000 7f3f876c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f801022b0 0x7f3f8019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f876c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8010c210 con 0x7f3f801018f0 2024-03-16T09:43:40.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f876c0640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8010c090 con 0x7f3f80100700 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f876c0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8010c390 con 0x7f3f801022b0 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f8545c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80100700 0x7f3f80196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55820/0 (socket says 172.21.15.77:55820) 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2092854359 learned_addr learned my addr 172.21.15.77:0/2092854359 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f8545c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80100700 0x7f3f80196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:40524/0 (socket says 172.21.15.77:40524) 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f801022b0 0x7f3f8019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2092854359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f801022b0 msgr2=0x7f3f8019de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f801022b0 0x7f3f8019de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2092854359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80100700 msgr2=0x7f3f80196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80100700 0x7f3f80196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2092854359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8019e4a0 con 0x7f3f801018f0 2024-03-16T09:43:40.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f801022b0 0x7f3f8019de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80196e60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3f7002c840 tx=0x7f3f70005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f767fc640 1 -- 172.21.15.77:0/2092854359 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70043020 con 0x7f3f801018f0 2024-03-16T09:43:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f767fc640 1 -- 172.21.15.77:0/2092854359 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f70004350 con 0x7f3f801018f0 2024-03-16T09:43:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8019e730 con 0x7f3f801018f0 2024-03-16T09:43:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f767fc640 1 -- 172.21.15.77:0/2092854359 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70003ae0 con 0x7f3f801018f0 2024-03-16T09:43:40.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.670+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8019ec80 con 0x7f3f801018f0 2024-03-16T09:43:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.672+0000 7f3f767fc640 1 -- 172.21.15.77:0/2092854359 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3f70046020 con 0x7f3f801018f0 2024-03-16T09:43:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 Processor -- start 2024-03-16T09:43:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 -- start start 2024-03-16T09:43:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f767fc640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3f6c0a28a0 0x7f3f6c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa7451970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7456a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa7459540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ffa73b4a40 con 0x55ffa7451550 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ffa735d920 con 0x55ffa74570b0 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8a384740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ffa73b0ae0 con 0x55ffa7451f40 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3f6c0a28a0 0x7f3f6c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8545c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa7451970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8545c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa7451970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55826/0 (socket says 172.21.15.77:55826) 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f84c5b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7456a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.673+0000 7f3f8545c640 1 -- 172.21.15.77:0/1044785558 learned_addr learned my addr 172.21.15.77:0/1044785558 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f767fc640 1 -- 172.21.15.77:0/2092854359 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3f700f0bd0 con 0x7f3f801018f0 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa7459540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f84c5b640 1 -- 172.21.15.77:0/1044785558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 msgr2=0x55ffa7459540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa7459540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f84c5b640 1 -- 172.21.15.77:0/1044785558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 msgr2=0x55ffa7451970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa7451970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f84c5b640 1 -- 172.21.15.77:0/1044785558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffa7375460 con 0x55ffa7451f40 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f8545c640 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa7451970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3f6c0a28a0 0x7f3f6c0a4d60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005fd0 tx=0x7f3f78009bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7456a40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f700bd510 tx=0x7f3f700035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f84a24f1b81ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f6b7fe640 1 -- 172.21.15.77:0/1044785558 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700d0340 con 0x55ffa7451f40 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f6b7fe640 1 -- 172.21.15.77:0/1044785558 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f700bdc80 con 0x55ffa7451f40 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.674+0000 7f3f6b7fe640 1 -- 172.21.15.77:0/1044785558 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700d2d30 con 0x55ffa7451f40 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 -- 172.21.15.77:0/1044785558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 msgr2=0x55ffa7456a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7456a40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f700bd510 tx=0x7f3f700035f0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 -- 172.21.15.77:0/1044785558 shutdown_connections 2024-03-16T09:43:40.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa7459540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7456a40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 --2- 172.21.15.77:0/1044785558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa7451970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 -- 172.21.15.77:0/1044785558 >> 172.21.15.77:0/1044785558 conn(0x55ffa744cac0 msgr2=0x55ffa744cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 -- 172.21.15.77:0/1044785558 shutdown_connections 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 -- 172.21.15.77:0/1044785558 wait complete. 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 -- start start 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.675+0000 7f3f8a384740 1 Processor -- start 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451550 0x55ffa745b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451f40 0x55ffa745b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa74629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8a384740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ffa7459e20 con 0x55ffa7451f40 2024-03-16T09:43:40.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8a384740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ffa7370d70 con 0x55ffa74570b0 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8a384740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ffa7459fa0 con 0x55ffa7451550 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8a384740 10 client.0 _ll_register_callbacks cb 0x55ffa70c8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451f40 0x55ffa745b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451f40 0x55ffa745b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55828/0 (socket says 172.21.15.77:55828) 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2908634300 learned_addr learned my addr 172.21.15.77:0/2908634300 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f8545c640 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451550 0x55ffa745b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa74629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2908634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451550 msgr2=0x55ffa745b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451550 0x55ffa745b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 msgr2=0x55ffa74629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa74629e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 -- 172.21.15.77:0/2908634300 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffa70c9210 con 0x55ffa7451f40 2024-03-16T09:43:40.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451f40 0x55ffa745b9a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3f70000c00 tx=0x7f3f700d06c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a19410ed7988c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.676+0000 7f3f68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f68ff9640 1 -- 172.21.15.77:0/2908634300 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700c9070 con 0x55ffa7451f40 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f68ff9640 1 -- 172.21.15.77:0/2908634300 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3f700d2f20 con 0x55ffa7451f40 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f700bdc80 con 0x55ffa7451f40 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f68ff9640 1 -- 172.21.15.77:0/2908634300 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3f700d0520 con 0x55ffa7451f40 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 10 client.15885 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ffa7195620 con 0x55ffa7451f40 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f68ff9640 1 -- 172.21.15.77:0/2908634300 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f700bdc80 con 0x55ffa7451f40 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f68ff9640 1 -- 172.21.15.77:0/2908634300 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3f70005490 con 0x55ffa7451f40 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 10 client.15885 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 10 client.15885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 20 client.15885 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:40.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa7466ed0 0x55ffa74692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 4 client.15885 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffa73c1f80 con 0x55ffa7466ed0 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.677+0000 7f3f8545c640 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa7466ed0 0x55ffa74692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.678+0000 7f3f8545c640 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa7466ed0 0x55ffa74692c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbf5e944ab779188 server_cookie=c67cbe3335666fd4 in_seq=0 out_seq=0 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.678+0000 7f3f68ff9640 10 client.15885 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.678+0000 7f3f68ff9640 1 -- 172.21.15.77:0/2908634300 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ffa73c1f80 con 0x55ffa7466ed0 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.678+0000 7f3f68ff9640 10 client.15885 handle_command_reply: tid=0 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.678+0000 7f3f8a384740 1 client.15885 shutdown 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451f40 msgr2=0x55ffa745b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451f40 0x55ffa745b9a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3f70000c00 tx=0x7f3f700d06c0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 shutdown_connections 2024-03-16T09:43:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa7466ed0 0x55ffa74692c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa74570b0 0x55ffa74629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451f40 0x55ffa745b9a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f68ff9640 0 client.15885 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 --2- 172.21.15.77:0/2908634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451550 0x55ffa745b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 >> 172.21.15.77:0/2908634300 conn(0x55ffa744cac0 msgr2=0x55ffa7449bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 shutdown_connections 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 1 -- 172.21.15.77:0/2908634300 wait complete. 2024-03-16T09:43:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.679+0000 7f3f8a384740 20 client.15885 trim_cache size 0 max 16384 2024-03-16T09:43:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 Processor -- start 2024-03-16T09:43:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 -- start start 2024-03-16T09:43:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa746eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451f40 0x55ffa74629a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa74570b0 0x55ffa7449320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801a9140 con 0x55ffa7451550 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801a6a10 con 0x55ffa7451f40 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.693+0000 7f3f8a384740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801a4860 con 0x55ffa74570b0 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa746eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa746eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55842/0 (socket says 172.21.15.77:55842) 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 -- 172.21.15.77:0/3909602270 learned_addr learned my addr 172.21.15.77:0/3909602270 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa74570b0 0x55ffa7449320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451f40 0x55ffa74629a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 -- 172.21.15.77:0/3909602270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa74570b0 msgr2=0x55ffa7449320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa74570b0 0x55ffa7449320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 -- 172.21.15.77:0/3909602270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451f40 msgr2=0x55ffa74629a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451f40 0x55ffa74629a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 -- 172.21.15.77:0/3909602270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffa74698a0 con 0x55ffa7451550 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f8545c640 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa746eb30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c009e50 tx=0x7f3f7c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e1a8c6e299aa21 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f6affd640 1 -- 172.21.15.77:0/3909602270 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c024020 con 0x55ffa7451550 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f6affd640 1 -- 172.21.15.77:0/3909602270 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f7c010e50 con 0x55ffa7451550 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.694+0000 7f3f6affd640 1 -- 172.21.15.77:0/3909602270 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c005020 con 0x55ffa7451550 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 -- 172.21.15.77:0/3909602270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 msgr2=0x55ffa746eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa746eb30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c009e50 tx=0x7f3f7c005f90 comp rx=0 tx=0).stop 2024-03-16T09:43:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 -- 172.21.15.77:0/3909602270 shutdown_connections 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa74570b0 0x55ffa7449320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451f40 0x55ffa74629a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3909602270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa7451550 0x55ffa746eb30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 -- 172.21.15.77:0/3909602270 >> 172.21.15.77:0/3909602270 conn(0x55ffa744cac0 msgr2=0x55ffa744fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 -- 172.21.15.77:0/3909602270 shutdown_connections 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 -- 172.21.15.77:0/3909602270 wait complete. 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 -- start start 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.695+0000 7f3f8a384740 1 Processor -- start 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451550 0x55ffa7460560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7460aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa74570b0 0x55ffa7461a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801a4ed0 con 0x55ffa74570b0 2024-03-16T09:43:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801a6340 con 0x55ffa7451550 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801a55a0 con 0x55ffa7451f40 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 10 client.0 _ll_register_callbacks cb 0x7f3f70009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa74570b0 0x55ffa7461a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa74570b0 0x55ffa7461a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55850/0 (socket says 172.21.15.77:55850) 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 -- 172.21.15.77:0/3411376637 learned_addr learned my addr 172.21.15.77:0/3411376637 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8545c640 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451550 0x55ffa7460560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f84c5b640 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7460aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 -- 172.21.15.77:0/3411376637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 msgr2=0x55ffa7460aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7460aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 -- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451550 msgr2=0x55ffa7460560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451550 0x55ffa7460560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 -- 172.21.15.77:0/3411376637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffa7462050 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f85c5d640 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa74570b0 0x55ffa7461a60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800edd0 tx=0x7f3f78004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678fd241082bbc43 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f68ff9640 1 -- 172.21.15.77:0/3411376637 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7802f060 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f68ff9640 1 -- 172.21.15.77:0/3411376637 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3f78011b50 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.696+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f7c010e50 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f68ff9640 1 -- 172.21.15.77:0/3411376637 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3f78011d50 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 10 client.15891 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ffa74cd640 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f68ff9640 1 -- 172.21.15.77:0/3411376637 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f7c010e50 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f68ff9640 1 -- 172.21.15.77:0/3411376637 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3f7801e070 con 0x55ffa74570b0 2024-03-16T09:43:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 10 client.15891 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 20 client.15891 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa74ce0a0 0x55ffa74d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 4 client.15891 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ffa73bc2d0 con 0x55ffa74ce0a0 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8545c640 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa74ce0a0 0x55ffa74d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f68ff9640 10 client.15891 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.697+0000 7f3f8545c640 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa74ce0a0 0x55ffa74d04c0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf1e1ee3a8866b3 server_cookie=e9c9fdc7f14475c2 in_seq=0 out_seq=0 2024-03-16T09:43:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f68ff9640 1 -- 172.21.15.77:0/3411376637 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ffa73bc2d0 con 0x55ffa74ce0a0 2024-03-16T09:43:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f68ff9640 10 client.15891 handle_command_reply: tid=0 2024-03-16T09:43:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 client.15891 shutdown 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa74570b0 msgr2=0x55ffa7461a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa74570b0 0x55ffa7461a60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800edd0 tx=0x7f3f78004ab0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 shutdown_connections 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffa74ce0a0 0x55ffa74d04c0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffa74570b0 0x55ffa7461a60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffa7451f40 0x55ffa7460aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 --2- 172.21.15.77:0/3411376637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffa7451550 0x55ffa7460560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f68ff9640 0 client.15891 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 >> 172.21.15.77:0/3411376637 conn(0x55ffa744cac0 msgr2=0x55ffa7449320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 shutdown_connections 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 1 -- 172.21.15.77:0/3411376637 wait complete. 2024-03-16T09:43:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.706+0000 7f3f8a384740 20 client.15891 trim_cache size 0 max 16384 2024-03-16T09:43:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:43:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:43:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:43:40.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3f6c0a28a0 msgr2=0x7f3f6c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3f6c0a28a0 0x7f3f6c0a4d60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005fd0 tx=0x7f3f78009bb0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 msgr2=0x7f3f80196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80196e60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3f7002c840 tx=0x7f3f70005e00 comp rx=0 tx=0).stop 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 shutdown_connections 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3f6c0a28a0 0x7f3f6c0a4d60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f801022b0 0x7f3f8019de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f801018f0 0x7f3f80196e60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 --2- 172.21.15.77:0/2092854359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80100700 0x7f3f80196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 >> 172.21.15.77:0/2092854359 conn(0x7f3f800fbc70 msgr2=0x7f3f80104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 shutdown_connections 2024-03-16T09:43:40.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:40.707+0000 7f3f876c0640 1 -- 172.21.15.77:0/2092854359 wait complete. 2024-03-16T09:43:41.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:40 smithi119 ceph-mon[35632]: pgmap v399: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 74 MiB/s wr, 28 op/s 2024-03-16T09:43:41.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:40 smithi077 ceph-mon[32171]: pgmap v399: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 74 MiB/s wr, 28 op/s 2024-03-16T09:43:41.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:40 smithi129 ceph-mon[35841]: pgmap v399: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 74 MiB/s wr, 28 op/s 2024-03-16T09:43:41.706 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:41.706 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 readers = 7488282.50 kB/sec 2024-03-16T09:43:41.706 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 readers = 7480156.82 kB/sec 2024-03-16T09:43:41.706 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 7488282.50 kB/sec 2024-03-16T09:43:41.706 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 7488282.50 kB/sec 2024-03-16T09:43:41.706 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 7488282.50 kB/sec 2024-03-16T09:43:41.706 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:43:43.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:42 smithi119 ceph-mon[35632]: pgmap v400: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 84 MiB/s wr, 33 op/s 2024-03-16T09:43:43.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:42 smithi077 ceph-mon[32171]: pgmap v400: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 84 MiB/s wr, 33 op/s 2024-03-16T09:43:43.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:42 smithi129 ceph-mon[35841]: pgmap v400: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 84 MiB/s wr, 33 op/s 2024-03-16T09:43:44.101 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:44.102 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 re-readers = 7572422.50 kB/sec 2024-03-16T09:43:44.102 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 re-readers = 7562370.79 kB/sec 2024-03-16T09:43:44.102 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 7572422.50 kB/sec 2024-03-16T09:43:44.102 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 7572422.50 kB/sec 2024-03-16T09:43:44.102 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 7572422.50 kB/sec 2024-03-16T09:43:44.102 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:43:44.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:43 smithi077 ceph-mon[32171]: pgmap v401: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 82 MiB/s wr, 29 op/s 2024-03-16T09:43:44.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:43 smithi129 ceph-mon[35841]: pgmap v401: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 82 MiB/s wr, 29 op/s 2024-03-16T09:43:44.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:43 smithi119 ceph-mon[35632]: pgmap v401: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 82 MiB/s wr, 29 op/s 2024-03-16T09:43:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:46 smithi119 ceph-mon[35632]: pgmap v402: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 74 MiB/s wr, 29 op/s 2024-03-16T09:43:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:46 smithi077 ceph-mon[32171]: pgmap v402: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 74 MiB/s wr, 29 op/s 2024-03-16T09:43:47.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:46 smithi129 ceph-mon[35841]: pgmap v402: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 74 MiB/s wr, 29 op/s 2024-03-16T09:43:49.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:48 smithi119 ceph-mon[35632]: pgmap v403: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 55 MiB/s wr, 23 op/s 2024-03-16T09:43:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:48 smithi077 ceph-mon[32171]: pgmap v403: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 55 MiB/s wr, 23 op/s 2024-03-16T09:43:49.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:48 smithi129 ceph-mon[35841]: pgmap v403: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 55 MiB/s wr, 23 op/s 2024-03-16T09:43:50.787 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:50.787 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:50.787 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:50.787 INFO:tasks.workunit.client.0.smithi077.stdout:iozone test complete. 2024-03-16T09:43:50.788 INFO:tasks.workunit.client.0.smithi077.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Iozone: Performance Test of File I/O 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Version $Revision: 3.494 $ 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Compiled for 64 bit mode. 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Build: linux-AMD64 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-16T09:43:50.854 INFO:tasks.workunit.client.0.smithi077.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Run began: Sat Mar 16 09:43:50 2024 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Include close in write timing 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Include fsync in write timing 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: File size set to 1048576 kB 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Record Size 1024 kB 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Output is in kBytes/sec 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Time Resolution = 0.000001 seconds. 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Processor cache size set to 1024 kBytes. 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Processor cache line size set to 32 bytes. 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: File stride size set to 17 * record size. 2024-03-16T09:43:50.855 INFO:tasks.workunit.client.0.smithi077.stdout: Throughput test with 1 process 2024-03-16T09:43:50.856 INFO:tasks.workunit.client.0.smithi077.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-03-16T09:43:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:50 smithi119 ceph-mon[35632]: pgmap v404: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 25 MiB/s wr, 14 op/s 2024-03-16T09:43:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:50 smithi077 ceph-mon[32171]: pgmap v404: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 25 MiB/s wr, 14 op/s 2024-03-16T09:43:51.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:50 smithi129 ceph-mon[35841]: pgmap v404: 97 pgs: 97 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 25 MiB/s wr, 14 op/s 2024-03-16T09:43:52.392 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:52.393 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 initial writers = 695934.75 kB/sec 2024-03-16T09:43:52.393 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 initial writers = 695847.43 kB/sec 2024-03-16T09:43:52.393 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 695934.75 kB/sec 2024-03-16T09:43:52.393 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 695934.75 kB/sec 2024-03-16T09:43:52.393 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 695934.75 kB/sec 2024-03-16T09:43:52.393 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:43:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:52 smithi119 ceph-mon[35632]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 20 MiB/s wr, 14 op/s 2024-03-16T09:43:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:52 smithi077 ceph-mon[32171]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 20 MiB/s wr, 14 op/s 2024-03-16T09:43:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:52 smithi129 ceph-mon[35841]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 20 MiB/s wr, 14 op/s 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 -- 172.21.15.77:0/3199924989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4100720 msgr2=0x7f70c4100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 --2- 172.21.15.77:0/3199924989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4100720 0x7f70c4100b40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009900 tx=0x7f70b402bc30 comp rx=0 tx=0).stop 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 -- 172.21.15.77:0/3199924989 shutdown_connections 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 --2- 172.21.15.77:0/3199924989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c41022d0 0x7f70c4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 --2- 172.21.15.77:0/3199924989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70c4101910 0x7f70c4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 --2- 172.21.15.77:0/3199924989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4100720 0x7f70c4100b40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 -- 172.21.15.77:0/3199924989 >> 172.21.15.77:0/3199924989 conn(0x7f70c40fbc90 msgr2=0x7f70c40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 -- 172.21.15.77:0/3199924989 shutdown_connections 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 -- 172.21.15.77:0/3199924989 wait complete. 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.796+0000 7f70c9787640 1 Processor -- start 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 -- start start 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70c4100720 0x7f70c406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4101910 0x7f70c419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c41022d0 0x7f70c419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70c410c220 con 0x7f70c4101910 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f70c410c0a0 con 0x7f70c4100720 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70c410c3a0 con 0x7f70c41022d0 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4101910 0x7f70c419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4101910 0x7f70c419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40026/0 (socket says 172.21.15.77:40026) 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 -- 172.21.15.77:0/4250306772 learned_addr learned my addr 172.21.15.77:0/4250306772 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70c4100720 0x7f70c406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c37fe640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c41022d0 0x7f70c419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 -- 172.21.15.77:0/4250306772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c41022d0 msgr2=0x7f70c419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c41022d0 0x7f70c419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 -- 172.21.15.77:0/4250306772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70c4100720 msgr2=0x7f70c406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70c4100720 0x7f70c406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 -- 172.21.15.77:0/4250306772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c419f590 con 0x7f70c4101910 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70bbfff640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4101910 0x7f70c419c660 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f70ac00e9f0 tx=0x7f70ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c0ff9640 1 -- 172.21.15.77:0/4250306772 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac00cd80 con 0x7f70c4101910 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c0ff9640 1 -- 172.21.15.77:0/4250306772 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70ac0108e0 con 0x7f70c4101910 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c419f880 con 0x7f70c4101910 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.797+0000 7f70c0ff9640 1 -- 172.21.15.77:0/4250306772 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac018920 con 0x7f70c4101910 2024-03-16T09:43:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.798+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c419fdd0 con 0x7f70c4101910 2024-03-16T09:43:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70c0ff9640 1 -- 172.21.15.77:0/4250306772 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f70ac010a90 con 0x7f70c4101910 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 Processor -- start 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 -- start start 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef369760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369d30 0x55a0ef36e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef371330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0ef2cea40 con 0x55a0ef369d30 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a0ef277920 con 0x55a0ef36eea0 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70cc374740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a0ef2caae0 con 0x55a0ef369340 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70c2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369d30 0x55a0ef36e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70c2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369d30 0x55a0ef36e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40042/0 (socket says 172.21.15.77:40042) 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef369760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1585370561 learned_addr learned my addr 172.21.15.77:0/1585370561 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70bbfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef371330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.800+0000 7f70c0ff9640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f70b00a28a0 0x7f70b00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c37fe640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f70b00a28a0 0x7f70b00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1585370561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 msgr2=0x55a0ef369760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/1585370561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef369760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1585370561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 msgr2=0x55a0ef371330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/1585370561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef371330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1585370561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0ef28f460 con 0x55a0ef369d30 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c0ff9640 1 -- 172.21.15.77:0/4250306772 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f70ac0cee10 con 0x7f70c4101910 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/1585370561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369d30 0x55a0ef36e830 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009900 tx=0x7f70b4009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b7d2211fac5bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70c37fe640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f70b00a28a0 0x7f70b00a4d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55a0ef36a590 tx=0x7f70bc004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70b97fa640 1 -- 172.21.15.77:0/1585370561 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4045020 con 0x55a0ef369d30 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70b97fa640 1 -- 172.21.15.77:0/1585370561 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70b402c9a0 con 0x55a0ef369d30 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.801+0000 7f70b97fa640 1 -- 172.21.15.77:0/1585370561 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b40046c0 con 0x55a0ef369d30 2024-03-16T09:43:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- 172.21.15.77:0/1585370561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369d30 msgr2=0x55a0ef36e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 --2- 172.21.15.77:0/1585370561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369d30 0x55a0ef36e830 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009900 tx=0x7f70b4009610 comp rx=0 tx=0).stop 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- 172.21.15.77:0/1585370561 shutdown_connections 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 --2- 172.21.15.77:0/1585370561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef371330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 --2- 172.21.15.77:0/1585370561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369d30 0x55a0ef36e830 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 --2- 172.21.15.77:0/1585370561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef369760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- 172.21.15.77:0/1585370561 >> 172.21.15.77:0/1585370561 conn(0x55a0ef3648b0 msgr2=0x55a0ef364cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- 172.21.15.77:0/1585370561 shutdown_connections 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- 172.21.15.77:0/1585370561 wait complete. 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- start start 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 Processor -- start 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef373b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef37ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0ef371df0 con 0x55a0ef369340 2024-03-16T09:43:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a0ef28ad70 con 0x55a0ef36eea0 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.802+0000 7f70cc374740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a0ef371f70 con 0x55a0ef369d30 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70cc374740 10 client.0 _ll_register_callbacks cb 0x55a0eefe2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40046/0 (socket says 172.21.15.77:40046) 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 -- 172.21.15.77:0/68107316 learned_addr learned my addr 172.21.15.77:0/68107316 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c37fe640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef37ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70bbfff640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef373b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 -- 172.21.15.77:0/68107316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 msgr2=0x55a0ef373b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef373b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 -- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 msgr2=0x55a0ef37ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef37ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 -- 172.21.15.77:0/68107316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0eefe3210 con 0x55a0ef369340 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70bbfff640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef373b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:43:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3735d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f70b402c170 tx=0x7f70b4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a88cf3b0d9fdaf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70aa7fc640 1 -- 172.21.15.77:0/68107316 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b404e020 con 0x55a0ef369340 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70aa7fc640 1 -- 172.21.15.77:0/68107316 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f70b40035f0 con 0x55a0ef369340 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70b402c9a0 con 0x55a0ef369340 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70aa7fc640 1 -- 172.21.15.77:0/68107316 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f70b4045070 con 0x55a0ef369340 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70cc374740 10 client.15903 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a0ef0af620 con 0x55a0ef369340 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.803+0000 7f70aa7fc640 1 -- 172.21.15.77:0/68107316 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70b402c9a0 con 0x55a0ef369340 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70aa7fc640 1 -- 172.21.15.77:0/68107316 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f70b4004d40 con 0x55a0ef369340 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70cc374740 10 client.15903 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70cc374740 10 client.15903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70cc374740 20 client.15903 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70cc374740 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef37f040 0x55a0ef381430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70cc374740 4 client.15903 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0ef2dbf80 con 0x55a0ef37f040 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70bbfff640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef37f040 0x55a0ef381430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70aa7fc640 10 client.15903 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.804+0000 7f70bbfff640 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef37f040 0x55a0ef381430 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a8a01284a7e7945 server_cookie=1e39411308da2acb in_seq=0 out_seq=0 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70aa7fc640 1 -- 172.21.15.77:0/68107316 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a0ef2dbf80 con 0x55a0ef37f040 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70aa7fc640 10 client.15903 handle_command_reply: tid=0 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 client.15903 shutdown 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 msgr2=0x55a0ef3735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3735d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f70b402c170 tx=0x7f70b4004040 comp rx=0 tx=0).stop 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 shutdown_connections 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef37f040 0x55a0ef381430 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef37ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70aa7fc640 0 client.15903 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef373b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 --2- 172.21.15.77:0/68107316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3735d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 >> 172.21.15.77:0/68107316 conn(0x55a0ef3648b0 msgr2=0x55a0ef363bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 shutdown_connections 2024-03-16T09:43:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 1 -- 172.21.15.77:0/68107316 wait complete. 2024-03-16T09:43:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.805+0000 7f70cc374740 20 client.15903 trim_cache size 0 max 16384 2024-03-16T09:43:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 Processor -- start 2024-03-16T09:43:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 -- start start 2024-03-16T09:43:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef35fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef369d30 0x55a0ef372bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef36eea0 0x55a0ef373130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41ab390 con 0x55a0ef36eea0 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41a8c60 con 0x55a0ef369d30 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70cc374740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41a6ab0 con 0x55a0ef369340 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef36eea0 0x55a0ef373130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef36eea0 0x55a0ef373130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40062/0 (socket says 172.21.15.77:40062) 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 -- 172.21.15.77:0/3494277493 learned_addr learned my addr 172.21.15.77:0/3494277493 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70bbfff640 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef369d30 0x55a0ef372bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef35fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 -- 172.21.15.77:0/3494277493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 msgr2=0x55a0ef35fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef35fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 -- 172.21.15.77:0/3494277493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef369d30 msgr2=0x55a0ef372bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef369d30 0x55a0ef372bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 -- 172.21.15.77:0/3494277493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0ef36b960 con 0x55a0ef36eea0 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70bbfff640 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef369d30 0x55a0ef372bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.820+0000 7f70c37fe640 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef36eea0 0x55a0ef373130 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f70bc012b50 tx=0x7f70bc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696f7bdb869cd458 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70b8ff9640 1 -- 172.21.15.77:0/3494277493 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc01a070 con 0x55a0ef36eea0 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70b8ff9640 1 -- 172.21.15.77:0/3494277493 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70bc011d60 con 0x55a0ef36eea0 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70b8ff9640 1 -- 172.21.15.77:0/3494277493 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc020650 con 0x55a0ef36eea0 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 -- 172.21.15.77:0/3494277493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef36eea0 msgr2=0x55a0ef373130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef36eea0 0x55a0ef373130 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f70bc012b50 tx=0x7f70bc009840 comp rx=0 tx=0).stop 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 -- 172.21.15.77:0/3494277493 shutdown_connections 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef36eea0 0x55a0ef373130 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef369d30 0x55a0ef372bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 --2- 172.21.15.77:0/3494277493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369340 0x55a0ef35fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 -- 172.21.15.77:0/3494277493 >> 172.21.15.77:0/3494277493 conn(0x55a0ef3648b0 msgr2=0x55a0ef3683a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 -- 172.21.15.77:0/3494277493 shutdown_connections 2024-03-16T09:43:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.821+0000 7f70cc374740 1 -- 172.21.15.77:0/3494277493 wait complete. 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 -- start start 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 Processor -- start 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3782b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef3787f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef379800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41a7120 con 0x55a0ef369340 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41a8590 con 0x55a0ef36eea0 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70c41a77f0 con 0x55a0ef369d30 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70cc374740 10 client.0 _ll_register_callbacks cb 0x7f70ac097ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3782b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3782b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40066/0 (socket says 172.21.15.77:40066) 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1391313236 learned_addr learned my addr 172.21.15.77:0/1391313236 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c37fe640 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef379800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70bbfff640 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef3787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1391313236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 msgr2=0x55a0ef3787f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef3787f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 msgr2=0x55a0ef379800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef379800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 -- 172.21.15.77:0/1391313236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0ef379df0 con 0x55a0ef369340 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70c2ffd640 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3782b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x55a0ef3603f0 tx=0x7f70b4004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8342923802133e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.822+0000 7f70aa7fc640 1 -- 172.21.15.77:0/1391313236 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b404e040 con 0x55a0ef369340 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70aa7fc640 1 -- 172.21.15.77:0/1391313236 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f70b402c8e0 con 0x55a0ef369340 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70bc011d60 con 0x55a0ef369340 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70aa7fc640 1 -- 172.21.15.77:0/1391313236 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f70b4005700 con 0x55a0ef369340 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 10 client.15909 fetch_fsmap learned FSMap version 21 2024-03-16T09:43:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a0ef3e9d50 con 0x55a0ef369340 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70aa7fc640 1 -- 172.21.15.77:0/1391313236 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70bc011d60 con 0x55a0ef369340 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70aa7fc640 1 -- 172.21.15.77:0/1391313236 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f70b40035f0 con 0x55a0ef369340 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 10 client.15909 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 10 client.15909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 20 client.15909 populate_metadata read hostname 'smithi077' 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef3ea7b0 0x55a0ef3ecba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 4 client.15909 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0ef2d62d0 con 0x55a0ef3ea7b0 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.823+0000 7f70bbfff640 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef3ea7b0 0x55a0ef3ecba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.824+0000 7f70aa7fc640 10 client.15909 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.824+0000 7f70bbfff640 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef3ea7b0 0x55a0ef3ecba0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d613b6557803e14b server_cookie=7263e96b3154f782 in_seq=0 out_seq=0 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70aa7fc640 1 -- 172.21.15.77:0/1391313236 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0ef2d62d0 con 0x55a0ef3ea7b0 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70aa7fc640 10 client.15909 handle_command_reply: tid=0 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 client.15909 shutdown 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 msgr2=0x55a0ef3782b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3782b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x55a0ef3603f0 tx=0x7f70b4004e80 comp rx=0 tx=0).stop 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 shutdown_connections 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55a0ef3ea7b0 0x55a0ef3ecba0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a0ef36eea0 0x55a0ef379800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0ef369d30 0x55a0ef3787f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70aa7fc640 0 client.15909 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 --2- 172.21.15.77:0/1391313236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0ef369340 0x55a0ef3782b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 >> 172.21.15.77:0/1391313236 conn(0x55a0ef3648b0 msgr2=0x55a0ef3683a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 shutdown_connections 2024-03-16T09:43:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.833+0000 7f70cc374740 1 -- 172.21.15.77:0/1391313236 wait complete. 2024-03-16T09:43:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70cc374740 20 client.15909 trim_cache size 0 max 16384 2024-03-16T09:43:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:43:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:43:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:43:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f70b00a28a0 msgr2=0x7f70b00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f70b00a28a0 0x7f70b00a4d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55a0ef36a590 tx=0x7f70bc004ab0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4101910 msgr2=0x7f70c419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4101910 0x7f70c419c660 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f70ac00e9f0 tx=0x7f70ac00eeb0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 shutdown_connections 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f70b00a28a0 0x7f70b00a4d60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c41022d0 0x7f70c419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70c4101910 0x7f70c419c660 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 --2- 172.21.15.77:0/4250306772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70c4100720 0x7f70c406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:43:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 >> 172.21.15.77:0/4250306772 conn(0x7f70c40fbc90 msgr2=0x7f70c4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:43:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 shutdown_connections 2024-03-16T09:43:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:43:53.834+0000 7f70c9787640 1 -- 172.21.15.77:0/4250306772 wait complete. 2024-03-16T09:43:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:54 smithi119 ceph-mon[35632]: pgmap v406: 97 pgs: 97 active+clean; 1.3 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 MiB/s wr, 14 op/s 2024-03-16T09:43:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:54 smithi077 ceph-mon[32171]: pgmap v406: 97 pgs: 97 active+clean; 1.3 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 MiB/s wr, 14 op/s 2024-03-16T09:43:55.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:54 smithi129 ceph-mon[35841]: pgmap v406: 97 pgs: 97 active+clean; 1.3 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 MiB/s wr, 14 op/s 2024-03-16T09:43:56.140 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:56.141 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 rewriters = 739147.38 kB/sec 2024-03-16T09:43:56.141 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 rewriters = 739043.79 kB/sec 2024-03-16T09:43:56.141 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 739147.38 kB/sec 2024-03-16T09:43:56.141 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 739147.38 kB/sec 2024-03-16T09:43:56.141 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 739147.38 kB/sec 2024-03-16T09:43:56.141 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:43:56.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:55 smithi077 ceph-mon[32171]: pgmap v407: 97 pgs: 97 active+clean; 1.3 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 63 MiB/s wr, 33 op/s 2024-03-16T09:43:56.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:56 smithi129 ceph-mon[35841]: pgmap v407: 97 pgs: 97 active+clean; 1.3 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 63 MiB/s wr, 33 op/s 2024-03-16T09:43:56.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:56 smithi119 ceph-mon[35632]: pgmap v407: 97 pgs: 97 active+clean; 1.3 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 63 MiB/s wr, 33 op/s 2024-03-16T09:43:58.572 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:43:58.573 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 readers = 6944717.50 kB/sec 2024-03-16T09:43:58.573 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 readers = 6934982.27 kB/sec 2024-03-16T09:43:58.573 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 6944717.50 kB/sec 2024-03-16T09:43:58.573 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 6944717.50 kB/sec 2024-03-16T09:43:58.573 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 6944717.50 kB/sec 2024-03-16T09:43:58.573 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:43:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:43:58 smithi119 ceph-mon[35632]: pgmap v408: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 142 MiB/s wr, 68 op/s 2024-03-16T09:43:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:43:58 smithi077 ceph-mon[32171]: pgmap v408: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 142 MiB/s wr, 68 op/s 2024-03-16T09:43:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:43:58 smithi129 ceph-mon[35841]: pgmap v408: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 142 MiB/s wr, 68 op/s 2024-03-16T09:44:00.871 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:00.871 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 re-readers = 7483249.50 kB/sec 2024-03-16T09:44:00.871 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 re-readers = 7473610.63 kB/sec 2024-03-16T09:44:00.871 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 7483249.50 kB/sec 2024-03-16T09:44:00.871 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 7483249.50 kB/sec 2024-03-16T09:44:00.871 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 7483249.50 kB/sec 2024-03-16T09:44:00.871 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 1048576.00 kB 2024-03-16T09:44:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:00 smithi119 ceph-mon[35632]: pgmap v409: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 159 MiB/s wr, 72 op/s 2024-03-16T09:44:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:00 smithi077 ceph-mon[32171]: pgmap v409: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 159 MiB/s wr, 72 op/s 2024-03-16T09:44:01.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:00 smithi129 ceph-mon[35841]: pgmap v409: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 159 MiB/s wr, 72 op/s 2024-03-16T09:44:03.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:44:03.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:44:03.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 -- 172.21.15.77:0/3103308928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 msgr2=0x7fd520108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 --2- 172.21.15.77:0/3103308928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 0x7fd520108b20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd518009be0 tx=0x7fd51802bef0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 -- 172.21.15.77:0/3103308928 shutdown_connections 2024-03-16T09:44:03.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 --2- 172.21.15.77:0/3103308928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 0x7fd520108b20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 --2- 172.21.15.77:0/3103308928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5201018d0 0x7fd520101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 --2- 172.21.15.77:0/3103308928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201006e0 0x7fd520100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 -- 172.21.15.77:0/3103308928 >> 172.21.15.77:0/3103308928 conn(0x7fd5200fbc70 msgr2=0x7fd5200fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 -- 172.21.15.77:0/3103308928 shutdown_connections 2024-03-16T09:44:03.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 -- 172.21.15.77:0/3103308928 wait complete. 2024-03-16T09:44:03.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.147+0000 7fd526128640 1 Processor -- start 2024-03-16T09:44:03.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 -- start start 2024-03-16T09:44:03.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5201006e0 0x7fd5201968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201018d0 0x7fd520196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 0x7fd52019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd52010c260 con 0x7fd5201018d0 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd52010c0e0 con 0x7fd5201006e0 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd52010c3e0 con 0x7fd520102290 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201018d0 0x7fd520196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201018d0 0x7fd520196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49908/0 (socket says 172.21.15.77:49908) 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 -- 172.21.15.77:0/2363315693 learned_addr learned my addr 172.21.15.77:0/2363315693 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5201006e0 0x7fd5201968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51ffff640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 0x7fd52019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 -- 172.21.15.77:0/2363315693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 msgr2=0x7fd52019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 0x7fd52019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 -- 172.21.15.77:0/2363315693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5201006e0 msgr2=0x7fd5201968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5201006e0 0x7fd5201968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 -- 172.21.15.77:0/2363315693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52019e500 con 0x7fd5201018d0 2024-03-16T09:44:03.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51effd640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201018d0 0x7fd520196e20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e9f0 tx=0x7fd51400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51cff9640 1 -- 172.21.15.77:0/2363315693 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51400cd80 con 0x7fd5201018d0 2024-03-16T09:44:03.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd51cff9640 1 -- 172.21.15.77:0/2363315693 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5140054d0 con 0x7fd5201018d0 2024-03-16T09:44:03.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.148+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52019e7f0 con 0x7fd5201018d0 2024-03-16T09:44:03.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.149+0000 7fd51cff9640 1 -- 172.21.15.77:0/2363315693 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514010490 con 0x7fd5201018d0 2024-03-16T09:44:03.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.149+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52019ecf0 con 0x7fd5201018d0 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.151+0000 7fd51cff9640 1 -- 172.21.15.77:0/2363315693 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd514005020 con 0x7fd5201018d0 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.151+0000 7fd528dea740 1 Processor -- start 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd528dea740 1 -- start start 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd528dea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2170 0x55b9914e2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd528dea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2b60 0x55b9914e7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd528dea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914ea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd528dea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b991447500 con 0x55b9914e2170 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd528dea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b9913efbe0 con 0x55b9914e7cd0 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd528dea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b9914435a0 con 0x55b9914e2b60 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2170 0x55b9914e2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2170 0x55b9914e2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49924/0 (socket says 172.21.15.77:49924) 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 -- 172.21.15.77:0/2046139646 learned_addr learned my addr 172.21.15.77:0/2046139646 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51effd640 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51cff9640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd5080a2800 0x7fd5080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2b60 0x55b9914e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd5080a2800 0x7fd5080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 -- 172.21.15.77:0/2046139646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2b60 msgr2=0x55b9914e7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2b60 0x55b9914e7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51cff9640 1 -- 172.21.15.77:0/2363315693 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd5140cedd0 con 0x7fd5201018d0 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 -- 172.21.15.77:0/2046139646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 msgr2=0x55b9914ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914ea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 -- 172.21.15.77:0/2046139646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9914034b0 con 0x55b9914e2170 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2b60 0x55b9914e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.152+0000 7fd51ffff640 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2170 0x55b9914e2590 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd518004af0 tx=0x7fd5180338b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3a224e1e5f878e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd505ffb640 1 -- 172.21.15.77:0/2046139646 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518051070 con 0x55b9914e2170 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd505ffb640 1 -- 172.21.15.77:0/2046139646 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd518033cb0 con 0x55b9914e2170 2024-03-16T09:44:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd505ffb640 1 -- 172.21.15.77:0/2046139646 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518004750 con 0x55b9914e2170 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd51ffff640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd5080a2800 0x7fd5080a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd51804e000 tx=0x7fd51804a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd528dea740 1 -- 172.21.15.77:0/2046139646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2170 msgr2=0x55b9914e2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd528dea740 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2170 0x55b9914e2590 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd518004af0 tx=0x7fd5180338b0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd528dea740 1 -- 172.21.15.77:0/2046139646 shutdown_connections 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd528dea740 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd528dea740 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2b60 0x55b9914e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd528dea740 1 --2- 172.21.15.77:0/2046139646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2170 0x55b9914e2590 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.153+0000 7fd528dea740 1 -- 172.21.15.77:0/2046139646 >> 172.21.15.77:0/2046139646 conn(0x55b9914dd6e0 msgr2=0x55b9914ddae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 -- 172.21.15.77:0/2046139646 shutdown_connections 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 -- 172.21.15.77:0/2046139646 wait complete. 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 -- start start 2024-03-16T09:44:03.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 Processor -- start 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914ec6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e2b60 0x55b9914ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e7cd0 0x55b9914f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b9914eac80 con 0x55b9914e7cd0 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b991403160 con 0x55b9914e2b60 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b9914eae00 con 0x55b9914e2170 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd528dea740 10 client.0 _ll_register_callbacks cb 0x55b9913e7150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e7cd0 0x55b9914f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e7cd0 0x55b9914f3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49930/0 (socket says 172.21.15.77:49930) 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.154+0000 7fd51ffff640 1 -- 172.21.15.77:0/1966682724 learned_addr learned my addr 172.21.15.77:0/1966682724 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914ec6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51effd640 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e2b60 0x55b9914ecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51ffff640 1 -- 172.21.15.77:0/1966682724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 msgr2=0x55b9914ec6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51ffff640 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914ec6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51ffff640 1 -- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e2b60 msgr2=0x55b9914ecc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51ffff640 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e2b60 0x55b9914ecc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51ffff640 1 -- 172.21.15.77:0/1966682724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9912257d0 con 0x55b9914e7cd0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd51ffff640 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e7cd0 0x55b9914f3c70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd5180049d0 tx=0x7fd518033a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669b612aaba48747 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd4feffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd4feffd640 1 -- 172.21.15.77:0/1966682724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518059070 con 0x55b9914e7cd0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd4feffd640 1 -- 172.21.15.77:0/1966682724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd518004450 con 0x55b9914e7cd0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd518033cb0 con 0x55b9914e7cd0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd4feffd640 1 -- 172.21.15.77:0/1966682724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd518065050 con 0x55b9914e7cd0 2024-03-16T09:44:03.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd528dea740 10 client.15921 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b99115b3f0 con 0x55b9914e7cd0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.155+0000 7fd4feffd640 1 -- 172.21.15.77:0/1966682724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd518033cb0 con 0x55b9914e7cd0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd4feffd640 1 -- 172.21.15.77:0/1966682724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd518034b10 con 0x55b9914e7cd0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd528dea740 10 client.15921 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd528dea740 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd528dea740 20 client.15921 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd528dea740 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b9914f7dd0 0x55b9914fa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd528dea740 4 client.15921 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b991454a40 con 0x55b9914f7dd0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b9914f7dd0 0x55b9914fa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd4feffd640 10 client.15921 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.156+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b9914f7dd0 0x55b9914fa1c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce4c5eeeb6cf090 server_cookie=9b3c042f157fef02 in_seq=0 out_seq=0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd4feffd640 1 -- 172.21.15.77:0/1966682724 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b991454a40 con 0x55b9914f7dd0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd4feffd640 10 client.15921 handle_command_reply: tid=0 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 client.15921 shutdown 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e7cd0 msgr2=0x55b9914f3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e7cd0 0x55b9914f3c70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd5180049d0 tx=0x7fd518033a90 comp rx=0 tx=0).stop 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 shutdown_connections 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b9914f7dd0 0x55b9914fa1c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e7cd0 0x55b9914f3c70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e2b60 0x55b9914ecc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd4feffd640 0 client.15921 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 --2- 172.21.15.77:0/1966682724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914ec6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 >> 172.21.15.77:0/1966682724 conn(0x55b9914dd6e0 msgr2=0x55b9914dc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 shutdown_connections 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 1 -- 172.21.15.77:0/1966682724 wait complete. 2024-03-16T09:44:03.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.157+0000 7fd528dea740 20 client.15921 trim_cache size 0 max 16384 2024-03-16T09:44:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 Processor -- start 2024-03-16T09:44:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 -- start start 2024-03-16T09:44:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b9914eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914dc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201aa3b0 con 0x55b9914e2b60 2024-03-16T09:44:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201a7c80 con 0x55b9914e7cd0 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd528dea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201a5ad0 con 0x55b9914e2170 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.174+0000 7fd51effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b9914eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b9914eb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49934/0 (socket says 172.21.15.77:49934) 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 -- 172.21.15.77:0/1744218356 learned_addr learned my addr 172.21.15.77:0/1744218356 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51ffff640 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914dc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 -- 172.21.15.77:0/1744218356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 msgr2=0x55b9914eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 -- 172.21.15.77:0/1744218356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 msgr2=0x55b9914dc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914dc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 -- 172.21.15.77:0/1744218356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9914fa530 con 0x55b9914e2b60 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd51effd640 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b9914eb710 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd5140ccc50 tx=0x7fd51409b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a528a7420b59d5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd5057fa640 1 -- 172.21.15.77:0/1744218356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51409b0e0 con 0x55b9914e2b60 2024-03-16T09:44:03.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd5057fa640 1 -- 172.21.15.77:0/1744218356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd514005310 con 0x55b9914e2b60 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.175+0000 7fd5057fa640 1 -- 172.21.15.77:0/1744218356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5140046c0 con 0x55b9914e2b60 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- 172.21.15.77:0/1744218356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 msgr2=0x55b9914eb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b9914eb710 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd5140ccc50 tx=0x7fd51409b680 comp rx=0 tx=0).stop 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- 172.21.15.77:0/1744218356 shutdown_connections 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b9914dc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b9914eb710 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 --2- 172.21.15.77:0/1744218356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b9914eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- 172.21.15.77:0/1744218356 >> 172.21.15.77:0/1744218356 conn(0x55b9914dd6e0 msgr2=0x55b9914ddac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- 172.21.15.77:0/1744218356 shutdown_connections 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- 172.21.15.77:0/1744218356 wait complete. 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- start start 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 Processor -- start 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b99155a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b99155c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b99155eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201a6140 con 0x55b9914e2b60 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201a75b0 con 0x55b9914e7cd0 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.176+0000 7fd528dea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201a6810 con 0x55b9914e2170 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd528dea740 10 client.0 _ll_register_callbacks cb 0x7fd51804f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b99155a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b99155eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b99155c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b99155eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:49210/0 (socket says 172.21.15.77:49210) 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 -- 172.21.15.77:0/516136606 learned_addr learned my addr 172.21.15.77:0/516136606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 -- 172.21.15.77:0/516136606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 msgr2=0x55b99155a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b99155a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 -- 172.21.15.77:0/516136606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 msgr2=0x55b99155c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b99155c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 -- 172.21.15.77:0/516136606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b99111b0f0 con 0x55b9914e7cd0 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b99155a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:03.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51effd640 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b99155c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd51ffff640 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b99155eb00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd518004aa0 tx=0x7fd5180338b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42ec07eca0499b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.177+0000 7fd4feffd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd4feffd640 1 -- 172.21.15.77:0/516136606 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518059040 con 0x55b9914e7cd0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd4feffd640 1 -- 172.21.15.77:0/516136606 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd518043ac0 con 0x55b9914e7cd0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b99144ed90 con 0x55b9914e7cd0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd4feffd640 1 -- 172.21.15.77:0/516136606 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd518051040 con 0x55b9914e7cd0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 10 client.25781 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b99140e9d0 con 0x55b9914e7cd0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd4feffd640 1 -- 172.21.15.77:0/516136606 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b99144ed90 con 0x55b9914e7cd0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd4feffd640 1 -- 172.21.15.77:0/516136606 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd5180344d0 con 0x55b9914e7cd0 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 10 client.25781 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 10 client.25781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 20 client.25781 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b991560930 0x55b991562d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 4 client.25781 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b9913cc540 con 0x55b991560930 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.178+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b991560930 0x55b991562d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd4feffd640 10 client.25781 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd51f7fe640 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b991560930 0x55b991562d50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a960df9f0871bbe4 server_cookie=3e7297ce30cf508c in_seq=0 out_seq=0 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd4feffd640 1 -- 172.21.15.77:0/516136606 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b9913cc540 con 0x55b991560930 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd4feffd640 10 client.25781 handle_command_reply: tid=0 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 client.25781 shutdown 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 msgr2=0x55b99155eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b99155eb00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd518004aa0 tx=0x7fd5180338b0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 shutdown_connections 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55b991560930 0x55b991562d50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b9914e7cd0 0x55b99155eb00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd4feffd640 0 client.25781 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b9914e2b60 0x55b99155c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 --2- 172.21.15.77:0/516136606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b9914e2170 0x55b99155a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 >> 172.21.15.77:0/516136606 conn(0x55b9914dd6e0 msgr2=0x55b9914eb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 shutdown_connections 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 1 -- 172.21.15.77:0/516136606 wait complete. 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.179+0000 7fd528dea740 20 client.25781 trim_cache size 0 max 16384 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T09:44:03.189 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:44:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd5080a2800 msgr2=0x7fd5080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd5080a2800 0x7fd5080a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd51804e000 tx=0x7fd51804a000 comp rx=0 tx=0).stop 2024-03-16T09:44:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201018d0 msgr2=0x7fd520196e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201018d0 0x7fd520196e20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e9f0 tx=0x7fd51400edf0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 shutdown_connections 2024-03-16T09:44:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd5080a2800 0x7fd5080a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd520102290 0x7fd52019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5201018d0 0x7fd520196e20 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 --2- 172.21.15.77:0/2363315693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5201006e0 0x7fd5201968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 >> 172.21.15.77:0/2363315693 conn(0x7fd5200fbc70 msgr2=0x7fd520104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 shutdown_connections 2024-03-16T09:44:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.180+0000 7fd526128640 1 -- 172.21.15.77:0/2363315693 wait complete. 2024-03-16T09:44:03.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:02 smithi119 ceph-mon[35632]: pgmap v410: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-03-16T09:44:03.193 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:02 smithi077 ceph-mon[32171]: pgmap v410: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-03-16T09:44:03.194 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T09:44:03.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:44:03.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 -- 172.21.15.77:0/1992299595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60301033c0 msgr2=0x7f6030109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 --2- 172.21.15.77:0/1992299595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60301033c0 0x7f6030109c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6028009be0 tx=0x7f602802bef0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 -- 172.21.15.77:0/1992299595 shutdown_connections 2024-03-16T09:44:03.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 --2- 172.21.15.77:0/1992299595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60301033c0 0x7f6030109c50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 --2- 172.21.15.77:0/1992299595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030102a00 0x7f6030102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 --2- 172.21.15.77:0/1992299595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6030101810 0x7f6030101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 -- 172.21.15.77:0/1992299595 >> 172.21.15.77:0/1992299595 conn(0x7f60300fcd80 msgr2=0x7f60300ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 -- 172.21.15.77:0/1992299595 shutdown_connections 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.292+0000 7f6038305640 1 -- 172.21.15.77:0/1992299595 wait complete. 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 Processor -- start 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 -- start start 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030101810 0x7f60301979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6030102a00 0x7f6030197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60301033c0 0x7f603019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f603010d320 con 0x7f60301033c0 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f603010d1a0 con 0x7f6030101810 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f6038305640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f603010d4a0 con 0x7f6030102a00 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f60368a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60301033c0 0x7f603019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f60368a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60301033c0 0x7f603019ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49958/0 (socket says 172.21.15.77:49958) 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f60368a2640 1 -- 172.21.15.77:0/262407994 learned_addr learned my addr 172.21.15.77:0/262407994 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f60358a0640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6030102a00 0x7f6030197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.293+0000 7f60360a1640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030101810 0x7f60301979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60368a2640 1 -- 172.21.15.77:0/262407994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6030102a00 msgr2=0x7f6030197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60368a2640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6030102a00 0x7f6030197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60368a2640 1 -- 172.21.15.77:0/262407994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030101810 msgr2=0x7f60301979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60368a2640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030101810 0x7f60301979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60368a2640 1 -- 172.21.15.77:0/262407994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603019f5e0 con 0x7f60301033c0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60360a1640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030101810 0x7f60301979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60368a2640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60301033c0 0x7f603019ef70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6028004af0 tx=0x7f6028033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60277fe640 1 -- 172.21.15.77:0/262407994 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6028043020 con 0x7f60301033c0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603019f870 con 0x7f60301033c0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60277fe640 1 -- 172.21.15.77:0/262407994 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f602802ca60 con 0x7f60301033c0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f60277fe640 1 -- 172.21.15.77:0/262407994 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6028034690 con 0x7f60301033c0 2024-03-16T09:44:03.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.294+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f603019fdc0 con 0x7f60301033c0 2024-03-16T09:44:03.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.295+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6030108bb0 con 0x7f60301033c0 2024-03-16T09:44:03.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.296+0000 7f60277fe640 1 -- 172.21.15.77:0/262407994 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6028046020 con 0x7f60301033c0 2024-03-16T09:44:03.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.296+0000 7f60277fe640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f601c0a2800 0x7f601c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.296+0000 7f60277fe640 1 -- 172.21.15.77:0/262407994 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f60280f1880 con 0x7f60301033c0 2024-03-16T09:44:03.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.297+0000 7f60360a1640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f601c0a2800 0x7f601c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.297+0000 7f60360a1640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f601c0a2800 0x7f601c0a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f602c0096c0 tx=0x7f602c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:02 smithi129 ceph-mon[35841]: pgmap v410: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 171 MiB/s wr, 76 op/s 2024-03-16T09:44:03.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.300+0000 7f60277fe640 1 -- 172.21.15.77:0/262407994 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f602804d020 con 0x7f60301033c0 2024-03-16T09:44:03.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.512+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f603002d2d0 con 0x7f60301033c0 2024-03-16T09:44:03.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.513+0000 7f60277fe640 1 -- 172.21.15.77:0/262407994 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f60280be930 con 0x7f60301033c0 2024-03-16T09:44:03.516 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:44:03.516 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:44:03.516 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:44:03.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f601c0a2800 msgr2=0x7f601c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f601c0a2800 0x7f601c0a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f602c0096c0 tx=0x7f602c009260 comp rx=0 tx=0).stop 2024-03-16T09:44:03.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60301033c0 msgr2=0x7f603019ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60301033c0 0x7f603019ef70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6028004af0 tx=0x7f6028033c90 comp rx=0 tx=0).stop 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 shutdown_connections 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f601c0a2800 0x7f601c0a4cc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60301033c0 0x7f603019ef70 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6030102a00 0x7f6030197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 --2- 172.21.15.77:0/262407994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030101810 0x7f60301979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 >> 172.21.15.77:0/262407994 conn(0x7f60300fcd80 msgr2=0x7f6030105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 shutdown_connections 2024-03-16T09:44:03.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.515+0000 7f6038305640 1 -- 172.21.15.77:0/262407994 wait complete. 2024-03-16T09:44:03.529 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 -- 172.21.15.77:0/1258527540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01006e0 msgr2=0x7f69e0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1258527540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01006e0 0x7f69e0100b00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f69dc009900 tx=0x7f69dc02bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 -- 172.21.15.77:0/1258527540 shutdown_connections 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1258527540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69e0102290 0x7f69e0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1258527540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69e01018d0 0x7f69e0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1258527540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01006e0 0x7f69e0100b00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 -- 172.21.15.77:0/1258527540 >> 172.21.15.77:0/1258527540 conn(0x7f69e00fbc70 msgr2=0x7f69e00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 -- 172.21.15.77:0/1258527540 shutdown_connections 2024-03-16T09:44:03.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.629+0000 7f69e78b5640 1 -- 172.21.15.77:0/1258527540 wait complete. 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 Processor -- start 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 -- start start 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69e01006e0 0x7f69e0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01018d0 0x7f69e0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69e0102290 0x7f69e019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69e010d390 con 0x7f69e01018d0 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f69e010d210 con 0x7f69e01006e0 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69e010d510 con 0x7f69e0102290 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01018d0 0x7f69e0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01018d0 0x7f69e0197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49986/0 (socket says 172.21.15.77:49986) 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 -- 172.21.15.77:0/1807777340 learned_addr learned my addr 172.21.15.77:0/1807777340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e5651640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69e01006e0 0x7f69e0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69e0102290 0x7f69e019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 -- 172.21.15.77:0/1807777340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69e0102290 msgr2=0x7f69e019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69e0102290 0x7f69e019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 -- 172.21.15.77:0/1807777340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69e01006e0 msgr2=0x7f69e0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69e01006e0 0x7f69e0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 -- 172.21.15.77:0/1807777340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e019f6d0 con 0x7f69e01018d0 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69e0102290 0x7f69e019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e4e50640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01018d0 0x7f69e0197ff0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f69d000d950 tx=0x7f69d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69d67fc640 1 -- 172.21.15.77:0/1807777340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0017030 con 0x7f69e01018d0 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e019f9c0 con 0x7f69e01018d0 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69d67fc640 1 -- 172.21.15.77:0/1807777340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d0004020 con 0x7f69e01018d0 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.630+0000 7f69d67fc640 1 -- 172.21.15.77:0/1807777340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00047a0 con 0x7f69e01018d0 2024-03-16T09:44:03.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.631+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e019ff60 con 0x7f69e01018d0 2024-03-16T09:44:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69d67fc640 1 -- 172.21.15.77:0/1807777340 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f69d00041d0 con 0x7f69e01018d0 2024-03-16T09:44:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69ea577740 1 Processor -- start 2024-03-16T09:44:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69ea577740 1 -- start start 2024-03-16T09:44:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69ea577740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc753cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69ea577740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc7541d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69ea577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7544840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69ea577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dcc74a1490 con 0x55dcc75423b0 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.633+0000 7f69ea577740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55dcc7449c90 con 0x55dcc753c850 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69ea577740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dcc749d530 con 0x55dcc753d240 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69d67fc640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f69cc0a2800 0x7f69cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7544840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7544840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49998/0 (socket says 172.21.15.77:49998) 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e5651640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc7541d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 -- 172.21.15.77:0/32249934 learned_addr learned my addr 172.21.15.77:0/32249934 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e5e52640 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc753cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f69cc0a2800 0x7f69cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69d67fc640 1 -- 172.21.15.77:0/1807777340 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f69d00ce5c0 con 0x7f69e01018d0 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 -- 172.21.15.77:0/32249934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 msgr2=0x55dcc7541d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc7541d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 -- 172.21.15.77:0/32249934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 msgr2=0x55dcc753cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc753cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 -- 172.21.15.77:0/32249934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcc7504500 con 0x55dcc75423b0 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f69cc0a2800 0x7f69cc0a4cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55dcc753daa0 tx=0x7f69d8009240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69e4e50640 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7544840 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f69d00a3180 tx=0x7f69d009a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58d335c9c1cebd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69cb7fe640 1 -- 172.21.15.77:0/32249934 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d009f060 con 0x55dcc75423b0 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69cb7fe640 1 -- 172.21.15.77:0/32249934 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d00a36b0 con 0x55dcc75423b0 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.634+0000 7f69cb7fe640 1 -- 172.21.15.77:0/32249934 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00ce0d0 con 0x55dcc75423b0 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 -- 172.21.15.77:0/32249934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 msgr2=0x55dcc7544840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7544840 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f69d00a3180 tx=0x7f69d009a870 comp rx=0 tx=0).stop 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 -- 172.21.15.77:0/32249934 shutdown_connections 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7544840 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc7541d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 --2- 172.21.15.77:0/32249934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc753cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 -- 172.21.15.77:0/32249934 >> 172.21.15.77:0/32249934 conn(0x55dcc7537dc0 msgr2=0x55dcc75381c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 -- 172.21.15.77:0/32249934 shutdown_connections 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 -- 172.21.15.77:0/32249934 wait complete. 2024-03-16T09:44:03.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 -- start start 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.635+0000 7f69ea577740 1 Processor -- start 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69ea577740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753c850 0x55dcc7546cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69ea577740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc7547210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69ea577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc754e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69ea577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dcc75452a0 con 0x55dcc75423b0 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69ea577740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55dcc745d230 con 0x55dcc753d240 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69ea577740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dcc7545420 con 0x55dcc753c850 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69ea577740 10 client.0 _ll_register_callbacks cb 0x55dcc7499010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc754e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e4e50640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc7547210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc754e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50000/0 (socket says 172.21.15.77:50000) 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 -- 172.21.15.77:0/1245687844 learned_addr learned my addr 172.21.15.77:0/1245687844 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5651640 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753c850 0x55dcc7546cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 -- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753c850 msgr2=0x55dcc7546cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753c850 0x55dcc7546cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 -- 172.21.15.77:0/1245687844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 msgr2=0x55dcc7547210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc7547210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 -- 172.21.15.77:0/1245687844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcc746c100 con 0x55dcc75423b0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e4e50640 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc7547210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc754e240 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f69d8017060 tx=0x7f69d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365f770991830051 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69c8ff9640 1 -- 172.21.15.77:0/1245687844 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8003970 con 0x55dcc75423b0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.636+0000 7f69c8ff9640 1 -- 172.21.15.77:0/1245687844 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f69d8003b10 con 0x55dcc75423b0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69d00a36b0 con 0x55dcc75423b0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69c8ff9640 1 -- 172.21.15.77:0/1245687844 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f69d8003d50 con 0x55dcc75423b0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 10 client.15945 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dcc71b53f0 con 0x55dcc75423b0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69c8ff9640 1 -- 172.21.15.77:0/1245687844 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d00a36b0 con 0x55dcc75423b0 2024-03-16T09:44:03.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69c8ff9640 1 -- 172.21.15.77:0/1245687844 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f69d801d040 con 0x55dcc75423b0 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 10 client.15945 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 10 client.15945 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 20 client.15945 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc7552ad0 0x55dcc7554f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 4 client.15945 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcc74ae9d0 con 0x55dcc7552ad0 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.637+0000 7f69e5651640 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc7552ad0 0x55dcc7554f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.638+0000 7f69c8ff9640 10 client.15945 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.638+0000 7f69e5651640 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc7552ad0 0x55dcc7554f30 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bb70caf7667a1f80 server_cookie=f50e24f69304a041 in_seq=0 out_seq=0 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.638+0000 7f69c8ff9640 1 -- 172.21.15.77:0/1245687844 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dcc74ae9d0 con 0x55dcc7552ad0 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.638+0000 7f69c8ff9640 10 client.15945 handle_command_reply: tid=0 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.638+0000 7f69ea577740 1 client.15945 shutdown 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.638+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 msgr2=0x55dcc754e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.638+0000 7f69ea577740 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc754e240 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f69d8017060 tx=0x7f69d8004a40 comp rx=0 tx=0).stop 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 shutdown_connections 2024-03-16T09:44:03.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc754e240 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc7552ad0 0x55dcc7554f30 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc7547210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 --2- 172.21.15.77:0/1245687844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753c850 0x55dcc7546cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69c8ff9640 0 client.15945 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 >> 172.21.15.77:0/1245687844 conn(0x55dcc7537dc0 msgr2=0x55dcc7536fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 shutdown_connections 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 1 -- 172.21.15.77:0/1245687844 wait complete. 2024-03-16T09:44:03.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.639+0000 7f69ea577740 20 client.15945 trim_cache size 0 max 16384 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 Processor -- start 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 -- start start 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc7546750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc754e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7536750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69e01a9310 con 0x55dcc75423b0 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f69e01a6be0 con 0x55dcc753c850 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69ea577740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69e01a4a30 con 0x55dcc753d240 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7536750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7536750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50004/0 (socket says 172.21.15.77:50004) 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 -- 172.21.15.77:0/1974843979 learned_addr learned my addr 172.21.15.77:0/1974843979 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5651640 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc7546750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e4e50640 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc754e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 -- 172.21.15.77:0/1974843979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 msgr2=0x55dcc754e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc754e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 -- 172.21.15.77:0/1974843979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 msgr2=0x55dcc7546750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc7546750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 -- 172.21.15.77:0/1974843979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcc727f7d0 con 0x55dcc75423b0 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e4e50640 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc754e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69e5e52640 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7536750 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f69d80098e0 tx=0x7f69d80099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21abca74a444115 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.652+0000 7f69caffd640 1 -- 172.21.15.77:0/1974843979 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8003970 con 0x55dcc75423b0 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69caffd640 1 -- 172.21.15.77:0/1974843979 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d8038070 con 0x55dcc75423b0 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69caffd640 1 -- 172.21.15.77:0/1974843979 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d801fd30 con 0x55dcc75423b0 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 -- 172.21.15.77:0/1974843979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 msgr2=0x55dcc7536750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7536750 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f69d80098e0 tx=0x7f69d80099b0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 -- 172.21.15.77:0/1974843979 shutdown_connections 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc75423b0 0x55dcc7536750 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc753d240 0x55dcc754e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 --2- 172.21.15.77:0/1974843979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753c850 0x55dcc7546750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 -- 172.21.15.77:0/1974843979 >> 172.21.15.77:0/1974843979 conn(0x55dcc7537dc0 msgr2=0x55dcc7546cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 -- 172.21.15.77:0/1974843979 shutdown_connections 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 -- 172.21.15.77:0/1974843979 wait complete. 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 -- start start 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.653+0000 7f69ea577740 1 Processor -- start 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc753c850 0x55dcc754ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc754b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc75423b0 0x55dcc75b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69e01a50a0 con 0x55dcc753c850 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f69e01a6510 con 0x55dcc753d240 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69e01a5770 con 0x55dcc75423b0 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 10 client.0 _ll_register_callbacks cb 0x7f69d8009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc753c850 0x55dcc754ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc753c850 0x55dcc754ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50020/0 (socket says 172.21.15.77:50020) 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 -- 172.21.15.77:0/888165346 learned_addr learned my addr 172.21.15.77:0/888165346 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e4e50640 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc754b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5e52640 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc75423b0 0x55dcc75b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 -- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc75423b0 msgr2=0x55dcc75b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc75423b0 0x55dcc75b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 -- 172.21.15.77:0/888165346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 msgr2=0x55dcc754b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc754b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 -- 172.21.15.77:0/888165346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcc754e3d0 con 0x55dcc753c850 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69e5651640 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc753c850 0x55dcc754ae30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f69dc005ab0 tx=0x7f69dc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd7b209d6754630 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69c8ff9640 1 -- 172.21.15.77:0/888165346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc04bd80 con 0x55dcc753c850 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69c8ff9640 1 -- 172.21.15.77:0/888165346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f69dc004130 con 0x55dcc753c850 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.654+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69d8038070 con 0x55dcc753c850 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69c8ff9640 1 -- 172.21.15.77:0/888165346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f69dc004330 con 0x55dcc753c850 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 10 client.15951 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dcc75b8360 con 0x55dcc753c850 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69c8ff9640 1 -- 172.21.15.77:0/888165346 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d8038070 con 0x55dcc753c850 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69c8ff9640 1 -- 172.21.15.77:0/888165346 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f69dc0046c0 con 0x55dcc753c850 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 10 client.15951 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 10 client.15951 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 20 client.15951 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc75b8d10 0x55dcc75bb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 4 client.15951 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dcc74a8d20 con 0x55dcc75b8d10 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.655+0000 7f69e4e50640 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc75b8d10 0x55dcc75bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69c8ff9640 10 client.15951 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69e4e50640 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc75b8d10 0x55dcc75bb130 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=66cd45c6420c2eed server_cookie=e324ea768caa24e2 in_seq=0 out_seq=0 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69c8ff9640 1 -- 172.21.15.77:0/888165346 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dcc74a8d20 con 0x55dcc75b8d10 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69c8ff9640 10 client.15951 handle_command_reply: tid=0 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 client.15951 shutdown 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc753c850 msgr2=0x55dcc754ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc753c850 0x55dcc754ae30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f69dc005ab0 tx=0x7f69dc004040 comp rx=0 tx=0).stop 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 shutdown_connections 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dcc75423b0 0x55dcc75b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x55dcc75b8d10 0x55dcc75bb130 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dcc753d240 0x55dcc754b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 --2- 172.21.15.77:0/888165346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcc753c850 0x55dcc754ae30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69c8ff9640 0 client.15951 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 >> 172.21.15.77:0/888165346 conn(0x55dcc7537dc0 msgr2=0x55dcc7546cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 shutdown_connections 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 1 -- 172.21.15.77:0/888165346 wait complete. 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69ea577740 20 client.15951 trim_cache size 0 max 16384 2024-03-16T09:44:03.665 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:44:03.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f69cc0a2800 msgr2=0x7f69cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f69cc0a2800 0x7f69cc0a4cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55dcc753daa0 tx=0x7f69d8009240 comp rx=0 tx=0).stop 2024-03-16T09:44:03.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01018d0 msgr2=0x7f69e0197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01018d0 0x7f69e0197ff0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f69d000d950 tx=0x7f69d000de10 comp rx=0 tx=0).stop 2024-03-16T09:44:03.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 shutdown_connections 2024-03-16T09:44:03.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.656+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f69cc0a2800 0x7f69cc0a4cc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.657+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69e0102290 0x7f69e019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.657+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69e01018d0 0x7f69e0197ff0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.657+0000 7f69e78b5640 1 --2- 172.21.15.77:0/1807777340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69e01006e0 0x7f69e0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.657+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 >> 172.21.15.77:0/1807777340 conn(0x7f69e00fbc70 msgr2=0x7f69e0104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.657+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 shutdown_connections 2024-03-16T09:44:03.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.657+0000 7f69e78b5640 1 -- 172.21.15.77:0/1807777340 wait complete. 2024-03-16T09:44:03.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T09:44:03.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- 172.21.15.77:0/2647908046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1018e0 msgr2=0x7f72ec101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 --2- 172.21.15.77:0/2647908046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1018e0 0x7f72ec101d60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009900 tx=0x7f72d402bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- 172.21.15.77:0/2647908046 shutdown_connections 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 --2- 172.21.15.77:0/2647908046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ec1022a0 0x7f72ec108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 --2- 172.21.15.77:0/2647908046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1018e0 0x7f72ec101d60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 --2- 172.21.15.77:0/2647908046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72ec1006f0 0x7f72ec100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- 172.21.15.77:0/2647908046 >> 172.21.15.77:0/2647908046 conn(0x7f72ec0fbc80 msgr2=0x7f72ec0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- 172.21.15.77:0/2647908046 shutdown_connections 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- 172.21.15.77:0/2647908046 wait complete. 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 Processor -- start 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- start start 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ec1006f0 0x7f72ec196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72ec1018e0 0x7f72ec196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1022a0 0x7f72ec19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec10c220 con 0x7f72ec1022a0 2024-03-16T09:44:03.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec10c0a0 con 0x7f72ec1006f0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.770+0000 7f72f1868640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec10c3a0 con 0x7f72ec1018e0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1022a0 0x7f72ec19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eaffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ec1006f0 0x7f72ec196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1022a0 0x7f72ec19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50046/0 (socket says 172.21.15.77:50046) 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 -- 172.21.15.77:0/4142479464 learned_addr learned my addr 172.21.15.77:0/4142479464 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72e3fff640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72ec1018e0 0x7f72ec196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 -- 172.21.15.77:0/4142479464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72ec1018e0 msgr2=0x7f72ec196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72ec1018e0 0x7f72ec196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 -- 172.21.15.77:0/4142479464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ec1006f0 msgr2=0x7f72ec196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ec1006f0 0x7f72ec196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 -- 172.21.15.77:0/4142479464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ec19e580 con 0x7f72ec1022a0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72eb7fe640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1022a0 0x7f72ec19df10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f72e400eb10 tx=0x7f72e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72f0866640 1 -- 172.21.15.77:0/4142479464 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4004b20 con 0x7f72ec1022a0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72f0866640 1 -- 172.21.15.77:0/4142479464 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72e4004cc0 con 0x7f72ec1022a0 2024-03-16T09:44:03.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ec19e870 con 0x7f72ec1022a0 2024-03-16T09:44:03.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72f0866640 1 -- 172.21.15.77:0/4142479464 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4010690 con 0x7f72ec1022a0 2024-03-16T09:44:03.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.771+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ec19ed40 con 0x7f72ec1022a0 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.773+0000 7f72f0866640 1 -- 172.21.15.77:0/4142479464 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f72e4005020 con 0x7f72ec1022a0 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.773+0000 7f72f444e740 1 Processor -- start 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.773+0000 7f72f444e740 1 -- start start 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f444e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x560292325e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f444e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x56029232af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f444e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x56029232da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f444e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560292288490 con 0x560292326450 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f444e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560292230c90 con 0x560292325a60 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f444e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560292284530 con 0x56029232b5c0 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x56029232af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f0866640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f72d80a2800 0x7f72d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x56029232af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50056/0 (socket says 172.21.15.77:50056) 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 -- 172.21.15.77:0/605623115 learned_addr learned my addr 172.21.15.77:0/605623115 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72e3fff640 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x560292325e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72e3fff640 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x56029232da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eb7fe640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f72d80a2800 0x7f72d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72f0866640 1 -- 172.21.15.77:0/4142479464 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f72e40cefe0 con 0x7f72ec1022a0 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 -- 172.21.15.77:0/605623115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 msgr2=0x56029232da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x56029232da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 -- 172.21.15.77:0/605623115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 msgr2=0x560292325e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x560292325e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 -- 172.21.15.77:0/605623115 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602922eb500 con 0x560292326450 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72e3fff640 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x56029232da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eaffd640 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x56029232af50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009ca0 tx=0x7f72dc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8ccab2626d4331 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72e1ffb640 1 -- 172.21.15.77:0/605623115 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc004220 con 0x560292326450 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72eb7fe640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f72d80a2800 0x7f72d80a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f72e4097df0 tx=0x7f72e4086040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72e1ffb640 1 -- 172.21.15.77:0/605623115 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72dc0043c0 con 0x560292326450 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.774+0000 7f72e1ffb640 1 -- 172.21.15.77:0/605623115 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc019890 con 0x560292326450 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 -- 172.21.15.77:0/605623115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 msgr2=0x56029232af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x56029232af50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009ca0 tx=0x7f72dc011810 comp rx=0 tx=0).stop 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 -- 172.21.15.77:0/605623115 shutdown_connections 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x56029232da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x56029232af50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 --2- 172.21.15.77:0/605623115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x560292325e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 -- 172.21.15.77:0/605623115 >> 172.21.15.77:0/605623115 conn(0x56029231edc0 msgr2=0x56029231f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 -- 172.21.15.77:0/605623115 shutdown_connections 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 -- 172.21.15.77:0/605623115 wait complete. 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.775+0000 7f72f444e740 1 -- start start 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 1 Processor -- start 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x56029232ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292326450 0x560292330450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56029232b5c0 0x560292337490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56029232e570 con 0x56029232b5c0 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560292244230 con 0x560292325a60 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56029232e6f0 con 0x560292326450 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72f444e740 10 client.0 _ll_register_callbacks cb 0x560292280010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56029232b5c0 0x560292337490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56029232b5c0 0x560292337490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50070/0 (socket says 172.21.15.77:50070) 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 -- 172.21.15.77:0/3445111011 learned_addr learned my addr 172.21.15.77:0/3445111011 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72eaffd640 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292326450 0x560292330450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x56029232ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 -- 172.21.15.77:0/3445111011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292326450 msgr2=0x560292330450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292326450 0x560292330450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 -- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 msgr2=0x56029232ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x56029232ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 -- 172.21.15.77:0/3445111011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560292253100 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72e3fff640 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56029232b5c0 0x560292337490 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009340 tx=0x7f72d4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c519070cc7da04e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.776+0000 7f72caffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72caffd640 1 -- 172.21.15.77:0/3445111011 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4005480 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72caffd640 1 -- 172.21.15.77:0/3445111011 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f72d40046c0 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72dc0043c0 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72caffd640 1 -- 172.21.15.77:0/3445111011 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f72d4057050 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 10 client.15963 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560291f9c3f0 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72caffd640 1 -- 172.21.15.77:0/3445111011 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72dc0043c0 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72caffd640 1 -- 172.21.15.77:0/3445111011 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f72d404e070 con 0x56029232b5c0 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 10 client.15963 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 10 client.15963 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 20 client.15963 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56029233bcc0 0x56029233e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 4 client.15963 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:03.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602922959d0 con 0x56029233bcc0 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.777+0000 7f72eaffd640 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56029233bcc0 0x56029233e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.778+0000 7f72eaffd640 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56029233bcc0 0x56029233e130 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f961ea0f311768ec server_cookie=75f94234a6efc1ee in_seq=0 out_seq=0 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.778+0000 7f72caffd640 10 client.15963 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72caffd640 1 -- 172.21.15.77:0/3445111011 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602922959d0 con 0x56029233bcc0 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72caffd640 10 client.15963 handle_command_reply: tid=0 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 client.15963 shutdown 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56029232b5c0 msgr2=0x560292337490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56029232b5c0 0x560292337490 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009340 tx=0x7f72d4005d50 comp rx=0 tx=0).stop 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 shutdown_connections 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56029233bcc0 0x56029233e130 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56029232b5c0 0x560292337490 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292326450 0x560292330450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72caffd640 0 client.15963 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 --2- 172.21.15.77:0/3445111011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292325a60 0x56029232ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 >> 172.21.15.77:0/3445111011 conn(0x56029231edc0 msgr2=0x56029231dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 shutdown_connections 2024-03-16T09:44:03.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 1 -- 172.21.15.77:0/3445111011 wait complete. 2024-03-16T09:44:03.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.779+0000 7f72f444e740 20 client.15963 trim_cache size 0 max 16384 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 Processor -- start 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 -- start start 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292325a60 0x56029231d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x560292337450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56029232b5c0 0x56029231dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec1a91a0 con 0x560292326450 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec1a6a70 con 0x56029232b5c0 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.792+0000 7f72f444e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec1a48c0 con 0x560292325a60 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x560292337450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292325a60 0x56029231d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292325a60 0x56029231d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:55188/0 (socket says 172.21.15.77:55188) 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x560292337450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50074/0 (socket says 172.21.15.77:50074) 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e3fff640 1 -- 172.21.15.77:0/424118783 learned_addr learned my addr 172.21.15.77:0/424118783 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e3fff640 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56029232b5c0 0x56029231dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 -- 172.21.15.77:0/424118783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292325a60 msgr2=0x56029231d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292325a60 0x56029231d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 -- 172.21.15.77:0/424118783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56029232b5c0 msgr2=0x56029231dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56029232b5c0 0x56029231dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 -- 172.21.15.77:0/424118783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602920667d0 con 0x560292326450 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e3fff640 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56029232b5c0 0x56029231dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72eaffd640 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x560292337450 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x56029231fd30 tx=0x7f72dc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ed8872d97d03d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e17fa640 1 -- 172.21.15.77:0/424118783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc02f020 con 0x560292326450 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e17fa640 1 -- 172.21.15.77:0/424118783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72dc02b2c0 con 0x560292326450 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.793+0000 7f72e17fa640 1 -- 172.21.15.77:0/424118783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0053e0 con 0x560292326450 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 -- 172.21.15.77:0/424118783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 msgr2=0x560292337450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x560292337450 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x56029231fd30 tx=0x7f72dc011810 comp rx=0 tx=0).stop 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 -- 172.21.15.77:0/424118783 shutdown_connections 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56029232b5c0 0x56029231dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292326450 0x560292337450 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 --2- 172.21.15.77:0/424118783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560292325a60 0x56029231d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 -- 172.21.15.77:0/424118783 >> 172.21.15.77:0/424118783 conn(0x56029231edc0 msgr2=0x5602923278f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 -- 172.21.15.77:0/424118783 shutdown_connections 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 -- 172.21.15.77:0/424118783 wait complete. 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 -- start start 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.794+0000 7f72f444e740 1 Processor -- start 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292325a60 0x560292334a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292326450 0x560292334fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x560292335fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec1a4f30 con 0x560292325a60 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec1a63a0 con 0x560292326450 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec1a5600 con 0x56029232b5c0 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 10 client.0 _ll_register_callbacks cb 0x7f72d4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292325a60 0x560292334a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292325a60 0x560292334a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50088/0 (socket says 172.21.15.77:50088) 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 -- 172.21.15.77:0/261192337 learned_addr learned my addr 172.21.15.77:0/261192337 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72eaffd640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292326450 0x560292334fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x560292335fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 -- 172.21.15.77:0/261192337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 msgr2=0x560292335fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x560292335fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 -- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292326450 msgr2=0x560292334fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292326450 0x560292334fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 -- 172.21.15.77:0/261192337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602923365b0 con 0x560292325a60 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72eaffd640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292326450 0x560292334fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72e3fff640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292325a60 0x560292334a70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009340 tx=0x7f72d4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aab353fd1bd8182 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72caffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72caffd640 1 -- 172.21.15.77:0/261192337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4063060 con 0x560292325a60 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72caffd640 1 -- 172.21.15.77:0/261192337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f72d4003c90 con 0x560292325a60 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72dc02b2c0 con 0x560292325a60 2024-03-16T09:44:03.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72caffd640 1 -- 172.21.15.77:0/261192337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f72d4003e90 con 0x560292325a60 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 10 client.15969 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.795+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5602923a2000 con 0x560292325a60 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72caffd640 1 -- 172.21.15.77:0/261192337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72dc02b2c0 con 0x560292325a60 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72caffd640 1 -- 172.21.15.77:0/261192337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f72d404e040 con 0x560292325a60 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72f444e740 10 client.15969 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72f444e740 10 client.15969 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72f444e740 20 client.15969 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72f444e740 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5602923a2930 0x5602923a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72f444e740 4 client.15969 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56029228fd20 con 0x5602923a2930 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72eaffd640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5602923a2930 0x5602923a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72caffd640 10 client.15969 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72eaffd640 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5602923a2930 0x5602923a4d50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f790f62dbde48360 server_cookie=873b4feff10f41a2 in_seq=0 out_seq=0 2024-03-16T09:44:03.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72caffd640 1 -- 172.21.15.77:0/261192337 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56029228fd20 con 0x5602923a2930 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72caffd640 10 client.15969 handle_command_reply: tid=0 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.796+0000 7f72f444e740 1 client.15969 shutdown 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292325a60 msgr2=0x560292334a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292325a60 0x560292334a70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009340 tx=0x7f72d4005d50 comp rx=0 tx=0).stop 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 shutdown_connections 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5602923a2930 0x5602923a4d50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56029232b5c0 0x560292335fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560292326450 0x560292334fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 --2- 172.21.15.77:0/261192337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560292325a60 0x560292334a70 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72caffd640 0 client.15969 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 >> 172.21.15.77:0/261192337 conn(0x56029231edc0 msgr2=0x560292322020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 shutdown_connections 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 1 -- 172.21.15.77:0/261192337 wait complete. 2024-03-16T09:44:03.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f444e740 20 client.15969 trim_cache size 0 max 16384 2024-03-16T09:44:03.807 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:44:03.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f72d80a2800 msgr2=0x7f72d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f72d80a2800 0x7f72d80a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f72e4097df0 tx=0x7f72e4086040 comp rx=0 tx=0).stop 2024-03-16T09:44:03.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1022a0 msgr2=0x7f72ec19df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1022a0 0x7f72ec19df10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f72e400eb10 tx=0x7f72e400c2d0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 shutdown_connections 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f72d80a2800 0x7f72d80a4cc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1022a0 0x7f72ec19df10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72ec1018e0 0x7f72ec196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 --2- 172.21.15.77:0/4142479464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ec1006f0 0x7f72ec196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 >> 172.21.15.77:0/4142479464 conn(0x7f72ec0fbc80 msgr2=0x7f72ec104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 shutdown_connections 2024-03-16T09:44:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.797+0000 7f72f1868640 1 -- 172.21.15.77:0/4142479464 wait complete. 2024-03-16T09:44:03.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.912+0000 7fcfff287640 1 -- 172.21.15.77:0/821338273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 msgr2=0x7fcff80fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.912+0000 7fcfff287640 1 --2- 172.21.15.77:0/821338273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff80fff00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009930 tx=0x7fcff402bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- 172.21.15.77:0/821338273 shutdown_connections 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 --2- 172.21.15.77:0/821338273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcff8103510 0x7fcff8109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 --2- 172.21.15.77:0/821338273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcff8102b50 0x7fcff8102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 --2- 172.21.15.77:0/821338273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff80fff00 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- 172.21.15.77:0/821338273 >> 172.21.15.77:0/821338273 conn(0x7fcff80fb410 msgr2=0x7fcff80fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- 172.21.15.77:0/821338273 shutdown_connections 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- 172.21.15.77:0/821338273 wait complete. 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 Processor -- start 2024-03-16T09:44:03.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- start start 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcff8102b50 0x7fcff8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcff8103510 0x7fcff819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcff810d390 con 0x7fcff80ffae0 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcff810d210 con 0x7fcff8102b50 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcfff287640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcff810d510 con 0x7fcff8103510 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcffd023640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.913+0000 7fcffd023640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff8197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50104/0 (socket says 172.21.15.77:50104) 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd023640 1 -- 172.21.15.77:0/192454514 learned_addr learned my addr 172.21.15.77:0/192454514 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffc822640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcff8102b50 0x7fcff8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd824640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcff8103510 0x7fcff819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd023640 1 -- 172.21.15.77:0/192454514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcff8103510 msgr2=0x7fcff819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd023640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcff8103510 0x7fcff819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd023640 1 -- 172.21.15.77:0/192454514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcff8102b50 msgr2=0x7fcff8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd023640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcff8102b50 0x7fcff8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd023640 1 -- 172.21.15.77:0/192454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff819f6d0 con 0x7fcff80ffae0 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffc822640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcff8102b50 0x7fcff8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcffd023640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff8197ab0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcff4000c00 tx=0x7fcff4005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcfee7fc640 1 -- 172.21.15.77:0/192454514 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4043020 con 0x7fcff80ffae0 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcfee7fc640 1 -- 172.21.15.77:0/192454514 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcff40043d0 con 0x7fcff80ffae0 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff819f960 con 0x7fcff80ffae0 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcfee7fc640 1 -- 172.21.15.77:0/192454514 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4003b80 con 0x7fcff80ffae0 2024-03-16T09:44:03.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.914+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff819feb0 con 0x7fcff80ffae0 2024-03-16T09:44:03.926 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.916+0000 7fcfee7fc640 1 -- 172.21.15.77:0/192454514 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcff4046020 con 0x7fcff80ffae0 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 Processor -- start 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 -- start start 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a09f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a09fbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a09fe840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5563a095b490 con 0x5563a09f7240 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5563a0903c90 con 0x5563a09f6850 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fd001e6a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5563a0957530 con 0x5563a09fc3b0 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcfee7fc640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffd023640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a09fbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffd023640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a09fbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50116/0 (socket says 172.21.15.77:50116) 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffd023640 1 -- 172.21.15.77:0/786302435 learned_addr learned my addr 172.21.15.77:0/786302435 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffd824640 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a09f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffc822640 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a09fe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffd824640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcfee7fc640 1 -- 172.21.15.77:0/192454514 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcff40f0ba0 con 0x7fcff80ffae0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffc822640 1 -- 172.21.15.77:0/786302435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 msgr2=0x5563a09f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffc822640 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a09f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffc822640 1 -- 172.21.15.77:0/786302435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 msgr2=0x5563a09fbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffc822640 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a09fbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.917+0000 7fcffc822640 1 -- 172.21.15.77:0/786302435 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a09be500 con 0x5563a09fc3b0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.918+0000 7fcffd824640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5563a09f7aa0 tx=0x7fcff0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.918+0000 7fcffc822640 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a09fe840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009fa0 tx=0x7fcfe8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adf575a4e42a3053 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.918+0000 7fcfe37fe640 1 -- 172.21.15.77:0/786302435 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8004230 con 0x5563a09fc3b0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.918+0000 7fcfe37fe640 1 -- 172.21.15.77:0/786302435 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe80043d0 con 0x5563a09fc3b0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.918+0000 7fcfe37fe640 1 -- 172.21.15.77:0/786302435 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe80035f0 con 0x5563a09fc3b0 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- 172.21.15.77:0/786302435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 msgr2=0x5563a09fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a09fe840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009fa0 tx=0x7fcfe8011810 comp rx=0 tx=0).stop 2024-03-16T09:44:03.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- 172.21.15.77:0/786302435 shutdown_connections 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a09fe840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a09fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 --2- 172.21.15.77:0/786302435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a09f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- 172.21.15.77:0/786302435 >> 172.21.15.77:0/786302435 conn(0x5563a09f1dc0 msgr2=0x5563a09f21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- 172.21.15.77:0/786302435 shutdown_connections 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- 172.21.15.77:0/786302435 wait complete. 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- start start 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 Processor -- start 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a01220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09fc3b0 0x5563a0a08250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5563a09ff360 con 0x5563a09f7240 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5563a0917230 con 0x5563a09fc3b0 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5563a09ff4e0 con 0x5563a09f6850 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.919+0000 7fd001e6a740 10 client.0 _ll_register_callbacks cb 0x5563a0953010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffc822640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a01220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffd023640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffd824640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09fc3b0 0x5563a0a08250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffd023640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:55230/0 (socket says 172.21.15.77:55230) 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffd023640 1 -- 172.21.15.77:0/2019635945 learned_addr learned my addr 172.21.15.77:0/2019635945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffc822640 1 -- 172.21.15.77:0/2019635945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09f6850 msgr2=0x5563a0a00ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffc822640 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffc822640 1 -- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09fc3b0 msgr2=0x5563a0a08250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffc822640 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09fc3b0 0x5563a0a08250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffc822640 1 -- 172.21.15.77:0/2019635945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a0926100 con 0x5563a09f7240 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffd023640 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffd824640 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09fc3b0 0x5563a0a08250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcffc822640 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a01220 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x5563a09f2d30 tx=0x7fcfe800fc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5202d89a29aac53c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcfe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:03.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/2019635945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8028020 con 0x5563a09f7240 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/2019635945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcfe8004bc0 con 0x5563a09f7240 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfe80043d0 con 0x5563a09f7240 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/2019635945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcfe8005c50 con 0x5563a09f7240 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fd001e6a740 10 client.15981 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5563a066f3f0 con 0x5563a09f7240 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.920+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/2019635945 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe80043d0 con 0x5563a09f7240 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/2019635945 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcfe8019790 con 0x5563a09f7240 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fd001e6a740 10 client.15981 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fd001e6a740 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fd001e6a740 20 client.15981 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fd001e6a740 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a0ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fd001e6a740 4 client.15981 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563a09689d0 con 0x5563a0a0c860 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fcffd023640 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a0ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fcffd023640 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a0ecb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff8b04ae01bb5e7 server_cookie=f1a626e0dff02a0c in_seq=0 out_seq=0 2024-03-16T09:44:03.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.921+0000 7fcfe0ff9640 10 client.15981 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/2019635945 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5563a09689d0 con 0x5563a0a0c860 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fcfe0ff9640 10 client.15981 handle_command_reply: tid=0 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 client.15981 shutdown 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 msgr2=0x5563a0a01220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a01220 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x5563a09f2d30 tx=0x7fcfe800fc40 comp rx=0 tx=0).stop 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 shutdown_connections 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a0ecb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09fc3b0 0x5563a0a08250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a01220 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fcfe0ff9640 0 client.15981 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 --2- 172.21.15.77:0/2019635945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 >> 172.21.15.77:0/2019635945 conn(0x5563a09f1dc0 msgr2=0x5563a09f0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 shutdown_connections 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.922+0000 7fd001e6a740 1 -- 172.21.15.77:0/2019635945 wait complete. 2024-03-16T09:44:03.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.923+0000 7fd001e6a740 20 client.15981 trim_cache size 0 max 16384 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 Processor -- start 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 -- start start 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f7240 0x5563a0a08210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a01220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81ab4e0 con 0x5563a09f6850 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81a8db0 con 0x5563a09f7240 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fd001e6a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81a6c00 con 0x5563a09fc3b0 2024-03-16T09:44:03.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50130/0 (socket says 172.21.15.77:50130) 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 -- 172.21.15.77:0/3016706593 learned_addr learned my addr 172.21.15.77:0/3016706593 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd824640 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a01220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffc822640 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f7240 0x5563a0a08210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 -- 172.21.15.77:0/3016706593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 msgr2=0x5563a0a01220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a01220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 -- 172.21.15.77:0/3016706593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f7240 msgr2=0x5563a0a08210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f7240 0x5563a0a08210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 -- 172.21.15.77:0/3016706593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a07397d0 con 0x5563a09f6850 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.936+0000 7fcffd023640 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fcff40b9ed0 tx=0x7fcff40f0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e9f0f775f20d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fcfe2ffd640 1 -- 172.21.15.77:0/3016706593 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff40bceb0 con 0x5563a09f6850 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fcfe2ffd640 1 -- 172.21.15.77:0/3016706593 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcff40bdc50 con 0x5563a09f6850 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fcfe2ffd640 1 -- 172.21.15.77:0/3016706593 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff40bb850 con 0x5563a09f6850 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 -- 172.21.15.77:0/3016706593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f6850 msgr2=0x5563a0a00ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fcff40b9ed0 tx=0x7fcff40f0040 comp rx=0 tx=0).stop 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 -- 172.21.15.77:0/3016706593 shutdown_connections 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a01220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f7240 0x5563a0a08210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 --2- 172.21.15.77:0/3016706593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f6850 0x5563a0a00ce0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 -- 172.21.15.77:0/3016706593 >> 172.21.15.77:0/3016706593 conn(0x5563a09f1dc0 msgr2=0x5563a09f21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 -- 172.21.15.77:0/3016706593 shutdown_connections 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 -- 172.21.15.77:0/3016706593 wait complete. 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.937+0000 7fd001e6a740 1 -- start start 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 1 Processor -- start 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a0a08270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a04d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81a7270 con 0x5563a09f7240 2024-03-16T09:44:03.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81a86e0 con 0x5563a09f6850 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcff81a7940 con 0x5563a09fc3b0 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fd001e6a740 10 client.0 _ll_register_callbacks cb 0x7fcfe8011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a04d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a04d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50142/0 (socket says 172.21.15.77:50142) 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 -- 172.21.15.77:0/4042441940 learned_addr learned my addr 172.21.15.77:0/4042441940 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffd023640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a0a08270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffd824640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 -- 172.21.15.77:0/4042441940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 msgr2=0x5563a0a052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 -- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 msgr2=0x5563a0a08270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a0a08270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 -- 172.21.15.77:0/4042441940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a0a70bd0 con 0x5563a09f7240 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffd023640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a0a08270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcffc822640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a04d10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8011850 tx=0x7fcfe800f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e6df960727bcf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.938+0000 7fcfe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:03.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/4042441940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe803c060 con 0x5563a09f7240 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/4042441940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcfe8019640 con 0x5563a09f7240 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcff40bdc50 con 0x5563a09f7240 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/4042441940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcfe8028040 con 0x5563a09f7240 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 10 client.15987 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5563a0a71ed0 con 0x5563a09f7240 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/4042441940 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcff40bdc50 con 0x5563a09f7240 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/4042441940 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcfe8036030 con 0x5563a09f7240 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 10 client.15987 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 20 client.15987 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a74890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 4 client.15987 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5563a0962d20 con 0x5563a0a0c860 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.939+0000 7fcffd023640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a74890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:03.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fcfe0ff9640 10 client.15987 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fcffd023640 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a74890 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6279420303d64b1f server_cookie=ed56b665fa6ab112 in_seq=0 out_seq=0 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fcfe0ff9640 1 -- 172.21.15.77:0/4042441940 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5563a0962d20 con 0x5563a0a0c860 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fcfe0ff9640 10 client.15987 handle_command_reply: tid=0 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 client.15987 shutdown 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 msgr2=0x5563a0a04d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a04d10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8011850 tx=0x7fcfe800f480 comp rx=0 tx=0).stop 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 shutdown_connections 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5563a0a0c860 0x5563a0a74890 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563a09fc3b0 0x5563a0a052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fcfe0ff9640 0 client.15987 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563a09f7240 0x5563a0a04d10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 --2- 172.21.15.77:0/4042441940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563a09f6850 0x5563a0a08270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 >> 172.21.15.77:0/4042441940 conn(0x5563a09f1dc0 msgr2=0x5563a09f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 shutdown_connections 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 1 -- 172.21.15.77:0/4042441940 wait complete. 2024-03-16T09:44:03.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.940+0000 7fd001e6a740 20 client.15987 trim_cache size 0 max 16384 2024-03-16T09:44:03.950 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe40a2800 msgr2=0x7fcfe40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5563a09f7aa0 tx=0x7fcff0009500 comp rx=0 tx=0).stop 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 msgr2=0x7fcff8197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff8197ab0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcff4000c00 tx=0x7fcff4005e60 comp rx=0 tx=0).stop 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 shutdown_connections 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcff8103510 0x7fcff819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcff8102b50 0x7fcff8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 --2- 172.21.15.77:0/192454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff80ffae0 0x7fcff8197ab0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 >> 172.21.15.77:0/192454514 conn(0x7fcff80fb410 msgr2=0x7fcff8105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 shutdown_connections 2024-03-16T09:44:03.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:03.941+0000 7fcfff287640 1 -- 172.21.15.77:0/192454514 wait complete. 2024-03-16T09:44:04.189 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:03 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/262407994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:44:04.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:03 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/262407994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:44:04.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:03 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/262407994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:44:04.955 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T09:44:04.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:44:04.955 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 -- 172.21.15.77:0/996924182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d301033c0 msgr2=0x7f8d30109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 --2- 172.21.15.77:0/996924182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d301033c0 0x7f8d30109c50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009ca0 tx=0x7f8d2802c0f0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 -- 172.21.15.77:0/996924182 shutdown_connections 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 --2- 172.21.15.77:0/996924182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d301033c0 0x7f8d30109c50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 --2- 172.21.15.77:0/996924182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d30102a00 0x7f8d30102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 --2- 172.21.15.77:0/996924182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d30101810 0x7f8d30101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 -- 172.21.15.77:0/996924182 >> 172.21.15.77:0/996924182 conn(0x7f8d300fcd80 msgr2=0x7f8d300ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 -- 172.21.15.77:0/996924182 shutdown_connections 2024-03-16T09:44:05.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 -- 172.21.15.77:0/996924182 wait complete. 2024-03-16T09:44:05.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 Processor -- start 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.072+0000 7f8d3710c640 1 -- start start 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d3710c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d30101810 0x7f8d301979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d3710c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d30102a00 0x7f8d30197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d3710c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d301033c0 0x7f8d3019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d3710c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3010d340 con 0x7f8d30101810 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d3710c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3010d1c0 con 0x7f8d301033c0 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d3710c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3010d4c0 con 0x7f8d30102a00 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d30101810 0x7f8d301979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d30101810 0x7f8d301979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50158/0 (socket says 172.21.15.77:50158) 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2122840472 learned_addr learned my addr 172.21.15.77:0/2122840472 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d30102a00 0x7f8d30197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d356a9640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d301033c0 0x7f8d3019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2122840472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d30102a00 msgr2=0x7f8d30197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d30102a00 0x7f8d30197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2122840472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d301033c0 msgr2=0x7f8d3019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d301033c0 0x7f8d3019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2122840472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3019f610 con 0x7f8d30101810 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d30101810 0x7f8d301979f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400e9c0 tx=0x7f8d2400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d2dffb640 1 -- 172.21.15.77:0/2122840472 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400cde0 con 0x7f8d30101810 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d2dffb640 1 -- 172.21.15.77:0/2122840472 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d240108e0 con 0x7f8d30101810 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3019f900 con 0x7f8d30101810 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.073+0000 7f8d2dffb640 1 -- 172.21.15.77:0/2122840472 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24018990 con 0x7f8d30101810 2024-03-16T09:44:05.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.074+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d3019fe50 con 0x7f8d30101810 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d2dffb640 1 -- 172.21.15.77:0/2122840472 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8d2401f020 con 0x7f8d30101810 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 Processor -- start 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 -- start start 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d2dffb640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8d200a2800 0x7f8d200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x56247062edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470633e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x562470636980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624705930d0 con 0x56247062e990 2024-03-16T09:44:05.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56247053bc70 con 0x56247062f380 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56247058f170 con 0x5624706344f0 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x56247062edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d356a9640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8d200a2800 0x7f8d200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x56247062edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50168/0 (socket says 172.21.15.77:50168) 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d34ea8640 1 -- 172.21.15.77:0/1646327511 learned_addr learned my addr 172.21.15.77:0/1646327511 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470633e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.076+0000 7f8d2dffb640 1 -- 172.21.15.77:0/2122840472 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8d240cec00 con 0x7f8d30101810 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d356a9640 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x562470636980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d34ea8640 1 -- 172.21.15.77:0/1646327511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 msgr2=0x562470636980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x562470636980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d34ea8640 1 -- 172.21.15.77:0/1646327511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 msgr2=0x562470633e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470633e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d34ea8640 1 -- 172.21.15.77:0/1646327511 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247036f7e0 con 0x56247062e990 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470633e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:05.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x56247062edb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8d240a3820 tx=0x7f8d2400a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b37bd616440782 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d356a9640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8d200a2800 0x7f8d200a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8d28006010 tx=0x7f8d2802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d1affd640 1 -- 172.21.15.77:0/1646327511 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d240adfb0 con 0x56247062e990 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d1affd640 1 -- 172.21.15.77:0/1646327511 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d24010e20 con 0x56247062e990 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.077+0000 7f8d1affd640 1 -- 172.21.15.77:0/1646327511 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d240a3d50 con 0x56247062e990 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1646327511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 msgr2=0x56247062edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x56247062edb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8d240a3820 tx=0x7f8d2400a440 comp rx=0 tx=0).stop 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1646327511 shutdown_connections 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x562470636980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470633e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1646327511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x56247062edb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1646327511 >> 172.21.15.77:0/1646327511 conn(0x5624705f6c50 msgr2=0x56247062a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1646327511 shutdown_connections 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1646327511 wait complete. 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 -- start start 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.078+0000 7f8d39dcb740 1 Processor -- start 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x562470638e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470639390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x5624706403d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624706374a0 con 0x56247062e990 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56247054f0d0 con 0x56247062f380 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562470637620 con 0x5624706344f0 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 10 client.0 _ll_register_callbacks cb 0x562470518760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x562470638e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x562470638e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50174/0 (socket says 172.21.15.77:50174) 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2358935319 learned_addr learned my addr 172.21.15.77:0/2358935319 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d356a9640 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x5624706403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470639390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2358935319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 msgr2=0x5624706403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x5624706403d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 msgr2=0x562470639390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470639390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2358935319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624703cff70 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x562470638e50 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8d24097f80 tx=0x7f8d2409bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864207c0190fd2c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2358935319 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2401d220 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2358935319 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8d24010a90 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.079+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5624705a0610 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2358935319 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8d2400a250 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 10 client.15999 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562470267170 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2358935319 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5624705a0610 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2358935319 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8d240ccbd0 con 0x56247062e990 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 10 client.15999 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 20 client.15999 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:05.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x562470644530 0x562470646920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 4 client.15999 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56247059a960 con 0x562470644530 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x562470644530 0x562470646920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d0ffff640 10 client.15999 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.080+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x562470644530 0x562470646920 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=223773e9c458a464 server_cookie=db3ef662184b115b in_seq=0 out_seq=0 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.081+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2358935319 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56247059a960 con 0x562470644530 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.081+0000 7f8d0ffff640 10 client.15999 handle_command_reply: tid=0 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.081+0000 7f8d39dcb740 1 client.15999 shutdown 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.081+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 msgr2=0x562470638e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.081+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x562470638e50 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8d24097f80 tx=0x7f8d2409bcd0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 shutdown_connections 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x562470644530 0x562470646920 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5624706344f0 0x5624706403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d0ffff640 0 client.15999 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062f380 0x562470639390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2358935319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062e990 0x562470638e50 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 >> 172.21.15.77:0/2358935319 conn(0x5624705f6c50 msgr2=0x5624706294e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 shutdown_connections 2024-03-16T09:44:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2358935319 wait complete. 2024-03-16T09:44:05.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.082+0000 7f8d39dcb740 20 client.15999 trim_cache size 0 max 16384 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 Processor -- start 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 -- start start 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062e990 0x56247062dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062f380 0x562470640360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624706344f0 0x562470638a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301ab490 con 0x5624706344f0 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301a8d60 con 0x56247062e990 2024-03-16T09:44:05.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301a6bb0 con 0x56247062f380 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d356a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624706344f0 0x562470638a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d356a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624706344f0 0x562470638a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50178/0 (socket says 172.21.15.77:50178) 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.099+0000 7f8d356a9640 1 -- 172.21.15.77:0/1514826180 learned_addr learned my addr 172.21.15.77:0/1514826180 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062e990 0x56247062dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062f380 0x562470640360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d356a9640 1 -- 172.21.15.77:0/1514826180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062f380 msgr2=0x562470640360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d356a9640 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062f380 0x562470640360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d356a9640 1 -- 172.21.15.77:0/1514826180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062e990 msgr2=0x56247062dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d356a9640 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062e990 0x56247062dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d356a9640 1 -- 172.21.15.77:0/1514826180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247062e190 con 0x5624706344f0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d356a9640 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624706344f0 0x562470638a60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8d28032420 tx=0x7f8d280324f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4731ceb1147231 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d1a7fc640 1 -- 172.21.15.77:0/1514826180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28039040 con 0x5624706344f0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d1a7fc640 1 -- 172.21.15.77:0/1514826180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d28055070 con 0x5624706344f0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.100+0000 7f8d1a7fc640 1 -- 172.21.15.77:0/1514826180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d280348a0 con 0x5624706344f0 2024-03-16T09:44:05.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1514826180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624706344f0 msgr2=0x562470638a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624706344f0 0x562470638a60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8d28032420 tx=0x7f8d280324f0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1514826180 shutdown_connections 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624706344f0 0x562470638a60 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062f380 0x562470640360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/1514826180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56247062e990 0x56247062dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1514826180 >> 172.21.15.77:0/1514826180 conn(0x5624705f6c50 msgr2=0x56247062d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1514826180 shutdown_connections 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- 172.21.15.77:0/1514826180 wait complete. 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- start start 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 Processor -- start 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062e990 0x56247063dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062f380 0x56247063e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5624706344f0 0x56247063f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301a7220 con 0x56247062f380 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301a8690 con 0x5624706344f0 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d301a78f0 con 0x56247062e990 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.101+0000 7f8d39dcb740 10 client.0 _ll_register_callbacks cb 0x7f8d2409b410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062f380 0x56247063e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062e990 0x56247063dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062f380 0x56247063e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50192/0 (socket says 172.21.15.77:50192) 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d34ea8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062e990 0x56247063dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:55300/0 (socket says 172.21.15.77:55300) 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d356a9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5624706344f0 0x56247063f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d34ea8640 1 -- 172.21.15.77:0/2785631514 learned_addr learned my addr 172.21.15.77:0/2785631514 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 -- 172.21.15.77:0/2785631514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062e990 msgr2=0x56247063dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062e990 0x56247063dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 -- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5624706344f0 msgr2=0x56247063f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5624706344f0 0x56247063f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 -- 172.21.15.77:0/2785631514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56247063f880 con 0x56247062f380 2024-03-16T09:44:05.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d356a9640 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5624706344f0 0x56247063f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d2ffff640 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062f380 0x56247063e280 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x562470646fb0 tx=0x7f8d1c010bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238653ed01c7313f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d0ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2785631514 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c023070 con 0x56247062f380 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2785631514 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8d1c004030 con 0x56247062f380 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d28055070 con 0x56247062f380 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2785631514 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8d1c02f050 con 0x56247062f380 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d39dcb740 10 client.16005 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5624706aa9b0 con 0x56247062f380 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.102+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2785631514 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d28055070 con 0x56247062f380 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2785631514 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8d1c0035f0 con 0x56247062f380 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d39dcb740 10 client.16005 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d39dcb740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d39dcb740 20 client.16005 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5624706ab3b0 0x5624706ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d39dcb740 4 client.16005 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562470518be0 con 0x5624706ab3b0 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5624706ab3b0 0x5624706ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d34ea8640 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5624706ab3b0 0x5624706ad7d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=297adcce5c822d0a server_cookie=f12348fac28f530a in_seq=0 out_seq=0 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.103+0000 7f8d0ffff640 10 client.16005 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d0ffff640 1 -- 172.21.15.77:0/2785631514 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562470518be0 con 0x5624706ab3b0 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d0ffff640 10 client.16005 handle_command_reply: tid=0 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 client.16005 shutdown 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062f380 msgr2=0x56247063e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062f380 0x56247063e280 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x562470646fb0 tx=0x7f8d1c010bc0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 shutdown_connections 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5624706ab3b0 0x5624706ad7d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5624706344f0 0x56247063f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56247062f380 0x56247063e280 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d0ffff640 0 client.16005 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 --2- 172.21.15.77:0/2785631514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56247062e990 0x56247063dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 >> 172.21.15.77:0/2785631514 conn(0x5624705f6c50 msgr2=0x562470638a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.104+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 shutdown_connections 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d39dcb740 1 -- 172.21.15.77:0/2785631514 wait complete. 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d39dcb740 20 client.16005 trim_cache size 0 max 16384 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "8cdff99d-7095-450f-b3c5-245e9925149b", 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T09:44:05.114 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:44:05.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8d200a2800 msgr2=0x7f8d200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8d200a2800 0x7f8d200a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8d28006010 tx=0x7f8d2802e040 comp rx=0 tx=0).stop 2024-03-16T09:44:05.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d30101810 msgr2=0x7f8d301979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d30101810 0x7f8d301979f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400e9c0 tx=0x7f8d2400ee80 comp rx=0 tx=0).stop 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 shutdown_connections 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8d200a2800 0x7f8d200a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d301033c0 0x7f8d3019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d30102a00 0x7f8d30197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 --2- 172.21.15.77:0/2122840472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d30101810 0x7f8d301979f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 >> 172.21.15.77:0/2122840472 conn(0x7f8d300fcd80 msgr2=0x7f8d30105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 shutdown_connections 2024-03-16T09:44:05.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.105+0000 7f8d3710c640 1 -- 172.21.15.77:0/2122840472 wait complete. 2024-03-16T09:44:05.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:44:05.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:44:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:04 smithi119 ceph-mon[35632]: pgmap v411: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 164 MiB/s wr, 73 op/s 2024-03-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 -- 172.21.15.77:0/2908172049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1029e0 msgr2=0x7f095c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 --2- 172.21.15.77:0/2908172049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1029e0 0x7f095c102e60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0948009900 tx=0x7f094802bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 -- 172.21.15.77:0/2908172049 shutdown_connections 2024-03-16T09:44:05.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 --2- 172.21.15.77:0/2908172049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f095c1033a0 0x7f095c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 --2- 172.21.15.77:0/2908172049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1029e0 0x7f095c102e60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 --2- 172.21.15.77:0/2908172049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f095c1017f0 0x7f095c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 -- 172.21.15.77:0/2908172049 >> 172.21.15.77:0/2908172049 conn(0x7f095c0fcd80 msgr2=0x7f095c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 -- 172.21.15.77:0/2908172049 shutdown_connections 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.220+0000 7f09630e8640 1 -- 172.21.15.77:0/2908172049 wait complete. 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 Processor -- start 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 -- start start 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1017f0 0x7f095c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f095c1029e0 0x7f095c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f095c1033a0 0x7f095c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f095c10d320 con 0x7f095c1017f0 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f095c10d1a0 con 0x7f095c1033a0 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f09630e8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f095c10d4a0 con 0x7f095c1029e0 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1017f0 0x7f095c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1017f0 0x7f095c075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50220/0 (socket says 172.21.15.77:50220) 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 -- 172.21.15.77:0/4281115613 learned_addr learned my addr 172.21.15.77:0/4281115613 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f095bfff640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f095c1029e0 0x7f095c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0961685640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f095c1033a0 0x7f095c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 -- 172.21.15.77:0/4281115613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f095c1029e0 msgr2=0x7f095c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f095c1029e0 0x7f095c073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 -- 172.21.15.77:0/4281115613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f095c1033a0 msgr2=0x7f095c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f095c1033a0 0x7f095c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 -- 172.21.15.77:0/4281115613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095c0749a0 con 0x7f095c1017f0 2024-03-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0961685640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f095c1033a0 0x7f095c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0960e84640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1017f0 0x7f095c075590 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f095000d950 tx=0x7f095000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0959ffb640 1 -- 172.21.15.77:0/4281115613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950017020 con 0x7f095c1017f0 2024-03-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.221+0000 7f0959ffb640 1 -- 172.21.15.77:0/4281115613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0950009c30 con 0x7f095c1017f0 2024-03-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.222+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f095c1a1b50 con 0x7f095c1017f0 2024-03-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.222+0000 7f0959ffb640 1 -- 172.21.15.77:0/4281115613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095000f6b0 con 0x7f095c1017f0 2024-03-16T09:44:05.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.222+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f095c1a2070 con 0x7f095c1017f0 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0959ffb640 1 -- 172.21.15.77:0/4281115613 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f095001a020 con 0x7f095c1017f0 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 Processor -- start 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0959ffb640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f094c0a2800 0x7f094c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 -- start start 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783ad590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783adb60 0x5587783b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0961685640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f094c0a2800 0x7f094c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783b5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558778312500 con 0x5587783adb60 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5587782babe0 con 0x5587783b2cd0 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0965dab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55877830e5a0 con 0x5587783ad170 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f0959ffb640 1 -- 172.21.15.77:0/4281115613 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f09500ce7c0 con 0x7f095c1017f0 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.224+0000 7f095bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783adb60 0x5587783b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783adb60 0x5587783b2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50224/0 (socket says 172.21.15.77:50224) 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 -- 172.21.15.77:0/2431507695 learned_addr learned my addr 172.21.15.77:0/2431507695 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f0961685640 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f0960e84640 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f0961685640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f094c0a2800 0x7f094c0a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0954005c40 tx=0x7f0954012b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 -- 172.21.15.77:0/2431507695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 msgr2=0x5587783ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 -- 172.21.15.77:0/2431507695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 msgr2=0x5587783b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 -- 172.21.15.77:0/2431507695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587782ce4b0 con 0x5587783adb60 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f095bfff640 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783adb60 0x5587783b2660 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0948009900 tx=0x7f0948009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843c42428f5581e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f0946ffd640 1 -- 172.21.15.77:0/2431507695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948045020 con 0x5587783adb60 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f0946ffd640 1 -- 172.21.15.77:0/2431507695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f094802c9a0 con 0x5587783adb60 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.225+0000 7f0946ffd640 1 -- 172.21.15.77:0/2431507695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09480055e0 con 0x5587783adb60 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 -- 172.21.15.77:0/2431507695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783adb60 msgr2=0x5587783b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783adb60 0x5587783b2660 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0948009900 tx=0x7f0948009610 comp rx=0 tx=0).stop 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 -- 172.21.15.77:0/2431507695 shutdown_connections 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783adb60 0x5587783b2660 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 --2- 172.21.15.77:0/2431507695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 -- 172.21.15.77:0/2431507695 >> 172.21.15.77:0/2431507695 conn(0x5587783a86e0 msgr2=0x5587783a8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 -- 172.21.15.77:0/2431507695 shutdown_connections 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 -- 172.21.15.77:0/2431507695 wait complete. 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 -- start start 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.226+0000 7f0965dab740 1 Processor -- start 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783b7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783beba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5587783b5c20 con 0x5587783ad170 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5587782ce160 con 0x5587783b2cd0 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5587783b5da0 con 0x5587783adb60 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 10 client.0 _ll_register_callbacks cb 0x5587782b2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783b7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50234/0 (socket says 172.21.15.77:50234) 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 -- 172.21.15.77:0/2647662047 learned_addr learned my addr 172.21.15.77:0/2647662047 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f095bfff640 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0961685640 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783beba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 -- 172.21.15.77:0/2647662047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 msgr2=0x5587783b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 -- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 msgr2=0x5587783beba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783beba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 -- 172.21.15.77:0/2647662047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587780f07d0 con 0x5587783ad170 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0960e84640 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783b7620 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0950097ae0 tx=0x7f09500bb1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b5cf9c5bcd8c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f093bfff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f093bfff640 1 -- 172.21.15.77:0/2647662047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095009f260 con 0x5587783ad170 2024-03-16T09:44:05.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f093bfff640 1 -- 172.21.15.77:0/2647662047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0950010a10 con 0x5587783ad170 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f094802c9a0 con 0x5587783ad170 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f093bfff640 1 -- 172.21.15.77:0/2647662047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f09500a38b0 con 0x5587783ad170 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 10 client.16017 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.227+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5587780263f0 con 0x5587783ad170 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f093bfff640 1 -- 172.21.15.77:0/2647662047 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f094802c9a0 con 0x5587783ad170 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f093bfff640 1 -- 172.21.15.77:0/2647662047 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0950024060 con 0x5587783ad170 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f0965dab740 10 client.16017 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f0965dab740 10 client.16017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f0965dab740 20 client.16017 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f0965dab740 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5587783c31a0 0x5587783c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f0965dab740 4 client.16017 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55877831fa40 con 0x5587783c31a0 2024-03-16T09:44:05.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f095bfff640 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5587783c31a0 0x5587783c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f095bfff640 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5587783c31a0 0x5587783c55f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f19a7e4ae54caca4 server_cookie=f18534397a3e4eff in_seq=0 out_seq=0 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.228+0000 7f093bfff640 10 client.16017 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f093bfff640 1 -- 172.21.15.77:0/2647662047 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55877831fa40 con 0x5587783c31a0 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f093bfff640 10 client.16017 handle_command_reply: tid=0 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 client.16017 shutdown 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 msgr2=0x5587783b7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783b7620 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0950097ae0 tx=0x7f09500bb1f0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 shutdown_connections 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5587783c31a0 0x5587783c55f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783beba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 --2- 172.21.15.77:0/2647662047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783b7620 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f093bfff640 0 client.16017 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 >> 172.21.15.77:0/2647662047 conn(0x5587783a86e0 msgr2=0x5587783a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 shutdown_connections 2024-03-16T09:44:05.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 1 -- 172.21.15.77:0/2647662047 wait complete. 2024-03-16T09:44:05.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.229+0000 7f0965dab740 20 client.16017 trim_cache size 0 max 16384 2024-03-16T09:44:05.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 Processor -- start 2024-03-16T09:44:05.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 -- start start 2024-03-16T09:44:05.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783beb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1ad680 con 0x5587783ad170 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1aaf50 con 0x5587783b2cd0 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.246+0000 7f0965dab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1a8da0 con 0x5587783adb60 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50242/0 (socket says 172.21.15.77:50242) 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 -- 172.21.15.77:0/1681344902 learned_addr learned my addr 172.21.15.77:0/1681344902 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f095bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783beb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0961685640 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 -- 172.21.15.77:0/1681344902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 msgr2=0x5587783beb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783beb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 -- 172.21.15.77:0/1681344902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 msgr2=0x5587783c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 -- 172.21.15.77:0/1681344902 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558777fe60f0 con 0x5587783ad170 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f0960e84640 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783ac050 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f09500a3a70 tx=0x7f095009b290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c48145d2f422a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f09467fc640 1 -- 172.21.15.77:0/1681344902 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950014070 con 0x5587783ad170 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f09467fc640 1 -- 172.21.15.77:0/1681344902 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09500662b0 con 0x5587783ad170 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.247+0000 7f09467fc640 1 -- 172.21.15.77:0/1681344902 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950097780 con 0x5587783ad170 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- 172.21.15.77:0/1681344902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 msgr2=0x5587783ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783ac050 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f09500a3a70 tx=0x7f095009b290 comp rx=0 tx=0).stop 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- 172.21.15.77:0/1681344902 shutdown_connections 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783b2cd0 0x5587783c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783adb60 0x5587783beb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 --2- 172.21.15.77:0/1681344902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783ad170 0x5587783ac050 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- 172.21.15.77:0/1681344902 >> 172.21.15.77:0/1681344902 conn(0x5587783a86e0 msgr2=0x5587783ab940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- 172.21.15.77:0/1681344902 shutdown_connections 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- 172.21.15.77:0/1681344902 wait complete. 2024-03-16T09:44:05.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- start start 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 Processor -- start 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783bc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783adb60 0x5587783bc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783b2cd0 0x5587783bd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1a9410 con 0x5587783b2cd0 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1aa880 con 0x5587783adb60 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.248+0000 7f0965dab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1a9ae0 con 0x5587783ad170 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0965dab740 10 client.0 _ll_register_callbacks cb 0x7f095009ab20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783b2cd0 0x5587783bd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0960e84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783bc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f095bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783adb60 0x5587783bc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783b2cd0 0x5587783bd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50244/0 (socket says 172.21.15.77:50244) 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 -- 172.21.15.77:0/4000879365 learned_addr learned my addr 172.21.15.77:0/4000879365 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 -- 172.21.15.77:0/4000879365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 msgr2=0x5587783bc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783bc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 -- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783adb60 msgr2=0x5587783bc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783adb60 0x5587783bc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 -- 172.21.15.77:0/4000879365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587782d99d0 con 0x5587783b2cd0 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f095bfff640 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783adb60 0x5587783bc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f093bfff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0961685640 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783b2cd0 0x5587783bd8d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f095401f730 tx=0x7f0954004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d452148b628b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f093bfff640 1 -- 172.21.15.77:0/4000879365 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954025040 con 0x5587783b2cd0 2024-03-16T09:44:05.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f093bfff640 1 -- 172.21.15.77:0/4000879365 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f095401e840 con 0x5587783b2cd0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09500662b0 con 0x5587783b2cd0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f093bfff640 1 -- 172.21.15.77:0/4000879365 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f095401ea40 con 0x5587783b2cd0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0965dab740 10 client.16023 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5587784294f0 con 0x5587783b2cd0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.249+0000 7f093bfff640 1 -- 172.21.15.77:0/4000879365 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09500662b0 con 0x5587783b2cd0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f093bfff640 1 -- 172.21.15.77:0/4000879365 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0954028030 con 0x5587783b2cd0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 10 client.16023 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 10 client.16023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 20 client.16023 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558778429fa0 0x55877842c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 4 client.16023 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558778319d90 con 0x558778429fa0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0960e84640 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558778429fa0 0x55877842c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:05.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0960e84640 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558778429fa0 0x55877842c3c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=999b5907e5793d6b server_cookie=288b172b6510a1f6 in_seq=0 out_seq=0 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f093bfff640 10 client.16023 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f093bfff640 1 -- 172.21.15.77:0/4000879365 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558778319d90 con 0x558778429fa0 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f093bfff640 10 client.16023 handle_command_reply: tid=0 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 client.16023 shutdown 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783b2cd0 msgr2=0x5587783bd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783b2cd0 0x5587783bd8d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f095401f730 tx=0x7f0954004ab0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 shutdown_connections 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x558778429fa0 0x55877842c3c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587783b2cd0 0x5587783bd8d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587783adb60 0x5587783bc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f093bfff640 0 client.16023 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.250+0000 7f0965dab740 1 --2- 172.21.15.77:0/4000879365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5587783ad170 0x5587783bc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 >> 172.21.15.77:0/4000879365 conn(0x5587783a86e0 msgr2=0x5587783ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 shutdown_connections 2024-03-16T09:44:05.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f0965dab740 1 -- 172.21.15.77:0/4000879365 wait complete. 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f0965dab740 20 client.16023 trim_cache size 0 max 16384 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: "8cdff99d-7095-450f-b3c5-245e9925149b": { 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "8cdff99d-7095-450f-b3c5-245e9925149b", 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:44:05.262 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:44:05.263 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:44:05.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f094c0a2800 msgr2=0x7f094c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f094c0a2800 0x7f094c0a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0954005c40 tx=0x7f0954012b50 comp rx=0 tx=0).stop 2024-03-16T09:44:05.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1017f0 msgr2=0x7f095c075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1017f0 0x7f095c075590 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f095000d950 tx=0x7f095000de10 comp rx=0 tx=0).stop 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 shutdown_connections 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f094c0a2800 0x7f094c0a4cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f095c1033a0 0x7f095c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f095c1029e0 0x7f095c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 --2- 172.21.15.77:0/4281115613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f095c1017f0 0x7f095c075590 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 >> 172.21.15.77:0/4281115613 conn(0x7f095c0fcd80 msgr2=0x7f095c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 shutdown_connections 2024-03-16T09:44:05.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:05.251+0000 7f09630e8640 1 -- 172.21.15.77:0/4281115613 wait complete. 2024-03-16T09:44:05.265 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:04 smithi077 ceph-mon[32171]: pgmap v411: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 164 MiB/s wr, 73 op/s 2024-03-16T09:44:05.265 INFO:tasks.cephfs.filesystem:scrub status for tag:8cdff99d-7095-450f-b3c5-245e9925149b - {'path': '/', 'tag': '8cdff99d-7095-450f-b3c5-245e9925149b', 'options': 'recursive,force'} 2024-03-16T09:44:05.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:04 smithi129 ceph-mon[35841]: pgmap v411: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 164 MiB/s wr, 73 op/s 2024-03-16T09:44:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:05 smithi119 ceph-mon[35632]: scrub queued for path: / 2024-03-16T09:44:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:05 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:05 smithi119 ceph-mon[35632]: scrub summary: active paths [/] 2024-03-16T09:44:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:05 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:06.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:05 smithi119 ceph-mon[35632]: scrub summary: idle 2024-03-16T09:44:06.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:05 smithi077 ceph-mon[32171]: scrub queued for path: / 2024-03-16T09:44:06.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:05 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:06.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:05 smithi077 ceph-mon[32171]: scrub summary: active paths [/] 2024-03-16T09:44:06.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:05 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:06.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:05 smithi077 ceph-mon[32171]: scrub summary: idle 2024-03-16T09:44:06.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:05 smithi129 ceph-mon[35841]: scrub queued for path: / 2024-03-16T09:44:06.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:05 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:06.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:05 smithi129 ceph-mon[35841]: scrub summary: active paths [/] 2024-03-16T09:44:06.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:05 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:06.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:05 smithi129 ceph-mon[35841]: scrub summary: idle 2024-03-16T09:44:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 -- 172.21.15.77:0/1064451896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b48101810 msgr2=0x7f9b48101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/1064451896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b48101810 0x7f9b48101c30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 -- 172.21.15.77:0/1064451896 shutdown_connections 2024-03-16T09:44:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/1064451896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b481033c0 0x7f9b48109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/1064451896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b48102a00 0x7f9b48102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/1064451896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b48101810 0x7f9b48101c30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 -- 172.21.15.77:0/1064451896 >> 172.21.15.77:0/1064451896 conn(0x7f9b480fcda0 msgr2=0x7f9b480ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 -- 172.21.15.77:0/1064451896 shutdown_connections 2024-03-16T09:44:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 -- 172.21.15.77:0/1064451896 wait complete. 2024-03-16T09:44:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.931+0000 7f9b4dde6640 1 Processor -- start 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b4dde6640 1 -- start start 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b4dde6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b48101810 0x7f9b48197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b4dde6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b48102a00 0x7f9b48197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b4dde6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b481033c0 0x7f9b4819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b4dde6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4810d310 con 0x7f9b481033c0 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b4dde6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4810d190 con 0x7f9b48101810 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b4dde6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4810d490 con 0x7f9b48102a00 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b481033c0 0x7f9b4819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b481033c0 0x7f9b4819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58596/0 (socket says 172.21.15.77:58596) 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 -- 172.21.15.77:0/574403511 learned_addr learned my addr 172.21.15.77:0/574403511 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b46ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b48102a00 0x7f9b48197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b477fe640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b48101810 0x7f9b48197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 -- 172.21.15.77:0/574403511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b48102a00 msgr2=0x7f9b48197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b48102a00 0x7f9b48197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 -- 172.21.15.77:0/574403511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b48101810 msgr2=0x7f9b48197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b48101810 0x7f9b48197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b47fff640 1 -- 172.21.15.77:0/574403511 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4819f5b0 con 0x7f9b481033c0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b48102a00 0x7f9b48197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.932+0000 7f9b477fe640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b48101810 0x7f9b48197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.933+0000 7f9b47fff640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b481033c0 0x7f9b4819ef40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000efd0 tx=0x7f9b4000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.933+0000 7f9b44ff9640 1 -- 172.21.15.77:0/574403511 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40003b70 con 0x7f9b481033c0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.933+0000 7f9b44ff9640 1 -- 172.21.15.77:0/574403511 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b40003d10 con 0x7f9b481033c0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.933+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4819f8a0 con 0x7f9b481033c0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.933+0000 7f9b44ff9640 1 -- 172.21.15.77:0/574403511 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40010980 con 0x7f9b481033c0 2024-03-16T09:44:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.933+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4819fda0 con 0x7f9b481033c0 2024-03-16T09:44:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b44ff9640 1 -- 172.21.15.77:0/574403511 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9b40004020 con 0x7f9b481033c0 2024-03-16T09:44:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 Processor -- start 2024-03-16T09:44:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 -- start start 2024-03-16T09:44:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e16340 0x5610e8e16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e1b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e1bea0 0x5610e8e1e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5610e8d7ba40 con 0x5610e8e16d30 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5610e8d24920 con 0x5610e8e1bea0 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.935+0000 7f9b509d4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5610e8d77ae0 con 0x5610e8e16340 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e1b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b46ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e16340 0x5610e8e16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b44ff9640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9b300a28a0 0x7f9b300a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e1b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58610/0 (socket says 172.21.15.77:58610) 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 -- 172.21.15.77:0/1456212868 learned_addr learned my addr 172.21.15.77:0/1456212868 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e1bea0 0x5610e8e1e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b47fff640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9b300a28a0 0x7f9b300a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 -- 172.21.15.77:0/1456212868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e16340 msgr2=0x5610e8e16760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e16340 0x5610e8e16760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 -- 172.21.15.77:0/1456212868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e1bea0 msgr2=0x5610e8e1e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e1bea0 0x5610e8e1e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 -- 172.21.15.77:0/1456212868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e8d3c460 con 0x5610e8e16d30 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b44ff9640 1 -- 172.21.15.77:0/574403511 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9b400ceeb0 con 0x7f9b481033c0 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b477fe640 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e1b830 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c0095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff370ef37294c4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b47fff640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9b300a28a0 0x7f9b300a4d60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9b400a3ad0 tx=0x7f9b4006a6c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b2dffb640 1 -- 172.21.15.77:0/1456212868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c045020 con 0x5610e8e16d30 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b2dffb640 1 -- 172.21.15.77:0/1456212868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b3c02cc90 con 0x5610e8e16d30 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.936+0000 7f9b2dffb640 1 -- 172.21.15.77:0/1456212868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c004920 con 0x5610e8e16d30 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 -- 172.21.15.77:0/1456212868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 msgr2=0x5610e8e1b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e1b830 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c0095f0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 -- 172.21.15.77:0/1456212868 shutdown_connections 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e1bea0 0x5610e8e1e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e1b830 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1456212868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e16340 0x5610e8e16760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 -- 172.21.15.77:0/1456212868 >> 172.21.15.77:0/1456212868 conn(0x5610e8e118b0 msgr2=0x5610e8e11cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 -- 172.21.15.77:0/1456212868 shutdown_connections 2024-03-16T09:44:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 -- 172.21.15.77:0/1456212868 wait complete. 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.937+0000 7f9b509d4740 1 -- start start 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 1 Processor -- start 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16340 0x5610e8e205e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16d30 0x5610e8e20b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e27b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5610e8e1edf0 con 0x5610e8e16340 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5610e8d37d70 con 0x5610e8e16d30 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5610e8e1ef70 con 0x5610e8e1bea0 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b509d4740 10 client.0 _ll_register_callbacks cb 0x5610e8a8fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16340 0x5610e8e205e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16d30 0x5610e8e20b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16340 0x5610e8e205e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58624/0 (socket says 172.21.15.77:58624) 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 -- 172.21.15.77:0/1227949229 learned_addr learned my addr 172.21.15.77:0/1227949229 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e27b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 -- 172.21.15.77:0/1227949229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 msgr2=0x5610e8e27b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e27b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 -- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16d30 msgr2=0x5610e8e20b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16d30 0x5610e8e20b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b46ffd640 1 -- 172.21.15.77:0/1227949229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e8a90210 con 0x5610e8e16340 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.938+0000 7f9b477fe640 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16d30 0x5610e8e20b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16340 0x5610e8e205e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9b380045b0 tx=0x7f9b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc3b30a2b318543 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b26ffd640 1 -- 172.21.15.77:0/1227949229 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38003d00 con 0x5610e8e16340 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b26ffd640 1 -- 172.21.15.77:0/1227949229 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9b38005020 con 0x5610e8e16340 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c02cc90 con 0x5610e8e16340 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b26ffd640 1 -- 172.21.15.77:0/1227949229 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9b38005220 con 0x5610e8e16340 2024-03-16T09:44:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 10 client.16035 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5610e8b5c620 con 0x5610e8e16340 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b26ffd640 1 -- 172.21.15.77:0/1227949229 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b3c02cc90 con 0x5610e8e16340 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b26ffd640 1 -- 172.21.15.77:0/1227949229 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9b380203f0 con 0x5610e8e16340 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 10 client.16035 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 10 client.16035 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 20 client.16035 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e2c440 0x5610e8e2e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 4 client.16035 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.939+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610e8d88f80 con 0x5610e8e2c440 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.940+0000 7f9b477fe640 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e2c440 0x5610e8e2e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.940+0000 7f9b477fe640 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e2c440 0x5610e8e2e8b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60264688e129e7d3 server_cookie=b0ea89afa8435de4 in_seq=0 out_seq=0 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.940+0000 7f9b26ffd640 10 client.16035 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b26ffd640 1 -- 172.21.15.77:0/1227949229 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5610e8d88f80 con 0x5610e8e2c440 2024-03-16T09:44:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b26ffd640 10 client.16035 handle_command_reply: tid=0 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 client.16035 shutdown 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16340 msgr2=0x5610e8e205e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16340 0x5610e8e205e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9b380045b0 tx=0x7f9b380046c0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 shutdown_connections 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e2c440 0x5610e8e2e8b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e27b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16d30 0x5610e8e20b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b26ffd640 0 client.16035 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 --2- 172.21.15.77:0/1227949229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16340 0x5610e8e205e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 >> 172.21.15.77:0/1227949229 conn(0x5610e8e118b0 msgr2=0x5610e8e10bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 shutdown_connections 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 1 -- 172.21.15.77:0/1227949229 wait complete. 2024-03-16T09:44:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.941+0000 7f9b509d4740 20 client.16035 trim_cache size 0 max 16384 2024-03-16T09:44:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 Processor -- start 2024-03-16T09:44:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 -- start start 2024-03-16T09:44:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e1fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e27b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e20900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9b481ab320 con 0x5610e8e16d30 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9b481a8bf0 con 0x5610e8e16340 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b509d4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b481a6a40 con 0x5610e8e1bea0 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e27b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e27b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58630/0 (socket says 172.21.15.77:58630) 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 -- 172.21.15.77:0/2709339289 learned_addr learned my addr 172.21.15.77:0/2709339289 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e1fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e20900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 -- 172.21.15.77:0/2709339289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 msgr2=0x5610e8e20900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e20900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 -- 172.21.15.77:0/2709339289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 msgr2=0x5610e8e1fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e1fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 -- 172.21.15.77:0/2709339289 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e8e0d310 con 0x5610e8e16d30 2024-03-16T09:44:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e1fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b477fe640 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e27b20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005ab0 tx=0x7f9b3c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66490025ec2f2e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b2d7fa640 1 -- 172.21.15.77:0/2709339289 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c045040 con 0x5610e8e16d30 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b2d7fa640 1 -- 172.21.15.77:0/2709339289 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b3c02cc90 con 0x5610e8e16d30 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.956+0000 7f9b2d7fa640 1 -- 172.21.15.77:0/2709339289 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0418a0 con 0x5610e8e16d30 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 -- 172.21.15.77:0/2709339289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 msgr2=0x5610e8e27b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e27b20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005ab0 tx=0x7f9b3c000bc0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 -- 172.21.15.77:0/2709339289 shutdown_connections 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e20900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e27b20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 --2- 172.21.15.77:0/2709339289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e1fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 -- 172.21.15.77:0/2709339289 >> 172.21.15.77:0/2709339289 conn(0x5610e8e118b0 msgr2=0x5610e8e201f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 -- 172.21.15.77:0/2709339289 shutdown_connections 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 -- 172.21.15.77:0/2709339289 wait complete. 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.957+0000 7f9b509d4740 1 -- start start 2024-03-16T09:44:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 1 Processor -- start 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e25510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e25a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e26a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9b481a70b0 con 0x5610e8e16d30 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9b481a8520 con 0x5610e8e16340 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b481a7780 con 0x5610e8e1bea0 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b509d4740 10 client.0 _ll_register_callbacks cb 0x7f9b38004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e25a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e25a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58632/0 (socket says 172.21.15.77:58632) 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 -- 172.21.15.77:0/3960934851 learned_addr learned my addr 172.21.15.77:0/3960934851 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e25510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e26a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 -- 172.21.15.77:0/3960934851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 msgr2=0x5610e8e26a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e26a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 -- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 msgr2=0x5610e8e25510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e25510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 -- 172.21.15.77:0/3960934851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e8e27050 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e25510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b477fe640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e25a50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c02c840 tx=0x7f9b3c02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98830ee82685b0b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.958+0000 7f9b26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b26ffd640 1 -- 172.21.15.77:0/3960934851 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c05a070 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b26ffd640 1 -- 172.21.15.77:0/3960934851 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9b3c003b00 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c02cc90 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b26ffd640 1 -- 172.21.15.77:0/3960934851 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c045040 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 10 client.16041 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5610e8e92ab0 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b26ffd640 1 -- 172.21.15.77:0/3960934851 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b3c02cc90 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b26ffd640 1 -- 172.21.15.77:0/3960934851 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9b3c054540 con 0x5610e8e16d30 2024-03-16T09:44:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 10 client.16041 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 10 client.16041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 20 client.16041 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e92fe0 0x5610e8e953d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 4 client.16041 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5610e8d832d0 con 0x5610e8e92fe0 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.959+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e92fe0 0x5610e8e953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.960+0000 7f9b26ffd640 10 client.16041 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.960+0000 7f9b46ffd640 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e92fe0 0x5610e8e953d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46a6f76f5ce2a227 server_cookie=905198a278b3a60d in_seq=0 out_seq=0 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b26ffd640 1 -- 172.21.15.77:0/3960934851 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5610e8d832d0 con 0x5610e8e92fe0 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b26ffd640 10 client.16041 handle_command_reply: tid=0 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 client.16041 shutdown 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 msgr2=0x5610e8e25a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e25a50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c02c840 tx=0x7f9b3c02c330 comp rx=0 tx=0).stop 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 shutdown_connections 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5610e8e92fe0 0x5610e8e953d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610e8e1bea0 0x5610e8e26a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b26ffd640 0 client.16041 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610e8e16d30 0x5610e8e25a50 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 --2- 172.21.15.77:0/3960934851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610e8e16340 0x5610e8e25510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 >> 172.21.15.77:0/3960934851 conn(0x5610e8e118b0 msgr2=0x5610e8e1fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 shutdown_connections 2024-03-16T09:44:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.968+0000 7f9b509d4740 1 -- 172.21.15.77:0/3960934851 wait complete. 2024-03-16T09:44:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b509d4740 20 client.16041 trim_cache size 0 max 16384 2024-03-16T09:44:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:44:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:44:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:44:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9b300a28a0 msgr2=0x7f9b300a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9b300a28a0 0x7f9b300a4d60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9b400a3ad0 tx=0x7f9b4006a6c0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b481033c0 msgr2=0x7f9b4819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b481033c0 0x7f9b4819ef40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000efd0 tx=0x7f9b4000c6a0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 shutdown_connections 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9b300a28a0 0x7f9b300a4d60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b481033c0 0x7f9b4819ef40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b48102a00 0x7f9b48197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 --2- 172.21.15.77:0/574403511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b48101810 0x7f9b48197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 >> 172.21.15.77:0/574403511 conn(0x7f9b480fcda0 msgr2=0x7f9b48105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 shutdown_connections 2024-03-16T09:44:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:06.969+0000 7f9b4dde6640 1 -- 172.21.15.77:0/574403511 wait complete. 2024-03-16T09:44:07.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:06 smithi119 ceph-mon[35632]: pgmap v412: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 144 MiB/s wr, 69 op/s 2024-03-16T09:44:07.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:06 smithi077 ceph-mon[32171]: pgmap v412: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 144 MiB/s wr, 69 op/s 2024-03-16T09:44:07.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:06 smithi129 ceph-mon[35841]: pgmap v412: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 144 MiB/s wr, 69 op/s 2024-03-16T09:44:07.350 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:07.350 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:07.350 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:07.350 INFO:tasks.workunit.client.0.smithi077.stdout:iozone test complete. 2024-03-16T09:44:07.351 INFO:tasks.workunit.client.0.smithi077.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-16T09:44:07.425 INFO:tasks.workunit.client.0.smithi077.stdout: Iozone: Performance Test of File I/O 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Version $Revision: 3.494 $ 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Compiled for 64 bit mode. 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Build: linux-AMD64 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-16T09:44:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Run began: Sat Mar 16 09:44:07 2024 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Include close in write timing 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Include fsync in write timing 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: File size set to 10485760 kB 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Record Size 1024 kB 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Output is in kBytes/sec 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Time Resolution = 0.000001 seconds. 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Processor cache size set to 1024 kBytes. 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Processor cache line size set to 32 bytes. 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: File stride size set to 17 * record size. 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Throughput test with 1 process 2024-03-16T09:44:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-03-16T09:44:09.051 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:08 smithi129 ceph-mon[35841]: pgmap v413: 97 pgs: 97 active+clean; 849 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 110 MiB/s wr, 56 op/s 2024-03-16T09:44:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:08 smithi119 ceph-mon[35632]: pgmap v413: 97 pgs: 97 active+clean; 849 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 110 MiB/s wr, 56 op/s 2024-03-16T09:44:09.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:08 smithi077 ceph-mon[32171]: pgmap v413: 97 pgs: 97 active+clean; 849 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 110 MiB/s wr, 56 op/s 2024-03-16T09:44:11.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:10 smithi119 ceph-mon[35632]: pgmap v414: 97 pgs: 97 active+clean; 893 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 52 MiB/s wr, 29 op/s 2024-03-16T09:44:11.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:10 smithi077 ceph-mon[32171]: pgmap v414: 97 pgs: 97 active+clean; 893 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 52 MiB/s wr, 29 op/s 2024-03-16T09:44:11.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:10 smithi129 ceph-mon[35841]: pgmap v414: 97 pgs: 97 active+clean; 893 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 52 MiB/s wr, 29 op/s 2024-03-16T09:44:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:12 smithi077 ceph-mon[32171]: pgmap v415: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 136 MiB/s wr, 68 op/s 2024-03-16T09:44:13.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:12 smithi129 ceph-mon[35841]: pgmap v415: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 136 MiB/s wr, 68 op/s 2024-03-16T09:44:13.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:12 smithi119 ceph-mon[35632]: pgmap v415: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 136 MiB/s wr, 68 op/s 2024-03-16T09:44:14.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:13 smithi077 ceph-mon[32171]: pgmap v416: 97 pgs: 97 active+clean; 2.4 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 206 MiB/s wr, 85 op/s 2024-03-16T09:44:14.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:13 smithi129 ceph-mon[35841]: pgmap v416: 97 pgs: 97 active+clean; 2.4 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 206 MiB/s wr, 85 op/s 2024-03-16T09:44:14.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:13 smithi119 ceph-mon[35632]: pgmap v416: 97 pgs: 97 active+clean; 2.4 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 206 MiB/s wr, 85 op/s 2024-03-16T09:44:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:44:15.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:14 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:44:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:14 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:44:15.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:15.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:14 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:16.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:15 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:44:16.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:15 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:44:16.261 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:15 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:16.261 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:15 smithi077 ceph-mon[32171]: pgmap v417: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 312 MiB/s wr, 113 op/s 2024-03-16T09:44:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:15 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:44:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:15 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:44:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:15 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:16.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:15 smithi129 ceph-mon[35841]: pgmap v417: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 312 MiB/s wr, 113 op/s 2024-03-16T09:44:16.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:15 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:44:16.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:15 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:44:16.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:15 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:44:16.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:15 smithi119 ceph-mon[35632]: pgmap v417: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 312 MiB/s wr, 113 op/s 2024-03-16T09:44:19.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:18 smithi119 ceph-mon[35632]: pgmap v418: 97 pgs: 97 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 496 MiB/s wr, 158 op/s 2024-03-16T09:44:19.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:18 smithi077 ceph-mon[32171]: pgmap v418: 97 pgs: 97 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 496 MiB/s wr, 158 op/s 2024-03-16T09:44:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:18 smithi129 ceph-mon[35841]: pgmap v418: 97 pgs: 97 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 496 MiB/s wr, 158 op/s 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.098+0000 7ff4f65d4640 1 -- 172.21.15.77:0/149211596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f0100700 msgr2=0x7ff4f0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.098+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/149211596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f0100700 0x7ff4f0100b20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009900 tx=0x7ff4e002bc50 comp rx=0 tx=0).stop 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.099+0000 7ff4f65d4640 1 -- 172.21.15.77:0/149211596 shutdown_connections 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.099+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/149211596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01022b0 0x7ff4f0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.099+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/149211596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f01018f0 0x7ff4f0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.099+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/149211596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f0100700 0x7ff4f0100b20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.099+0000 7ff4f65d4640 1 -- 172.21.15.77:0/149211596 >> 172.21.15.77:0/149211596 conn(0x7ff4f00fbc70 msgr2=0x7ff4f00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.099+0000 7ff4f65d4640 1 -- 172.21.15.77:0/149211596 shutdown_connections 2024-03-16T09:44:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.099+0000 7ff4f65d4640 1 -- 172.21.15.77:0/149211596 wait complete. 2024-03-16T09:44:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 Processor -- start 2024-03-16T09:44:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 -- start start 2024-03-16T09:44:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f0100700 0x7ff4f0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01018f0 0x7ff4f019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f01022b0 0x7ff4f01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f010c2a0 con 0x7ff4f01022b0 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f010c120 con 0x7ff4f01018f0 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.100+0000 7ff4f65d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f010c420 con 0x7ff4f0100700 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01018f0 0x7ff4f019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01018f0 0x7ff4f019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:45936/0 (socket says 172.21.15.77:45936) 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 -- 172.21.15.77:0/554079653 learned_addr learned my addr 172.21.15.77:0/554079653 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4effff640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f0100700 0x7ff4f0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 -- 172.21.15.77:0/554079653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f0100700 msgr2=0x7ff4f0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f0100700 0x7ff4f0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 -- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f01022b0 msgr2=0x7ff4f01a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f01022b0 0x7ff4f01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4ef7fe640 1 -- 172.21.15.77:0/554079653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f01a18d0 con 0x7ff4f01018f0 2024-03-16T09:44:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.101+0000 7ff4effff640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f0100700 0x7ff4f0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.102+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01018f0 0x7ff4f019a1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b450 tx=0x7ff4e400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.102+0000 7ff4ed7fa640 1 -- 172.21.15.77:0/554079653 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4015020 con 0x7ff4f01018f0 2024-03-16T09:44:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.102+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f01a1bc0 con 0x7ff4f01018f0 2024-03-16T09:44:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.102+0000 7ff4ed7fa640 1 -- 172.21.15.77:0/554079653 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4e4011d40 con 0x7ff4f01018f0 2024-03-16T09:44:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.102+0000 7ff4ed7fa640 1 -- 172.21.15.77:0/554079653 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e400f340 con 0x7ff4f01018f0 2024-03-16T09:44:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.103+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f01a20c0 con 0x7ff4f01018f0 2024-03-16T09:44:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.105+0000 7ff4ed7fa640 1 -- 172.21.15.77:0/554079653 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff4e4018020 con 0x7ff4f01018f0 2024-03-16T09:44:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 Processor -- start 2024-03-16T09:44:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 -- start start 2024-03-16T09:44:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4ed7fa640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff4d80a28a0 0x7ff4d80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cda340 0x565260cda760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260cdf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdfea0 0x565260ce2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565260c3fa40 con 0x565260cdfea0 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x565260be8920 con 0x565260cda340 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x565260c3bae0 con 0x565260cdad30 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.106+0000 7ff4ed7fa640 1 -- 172.21.15.77:0/554079653 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff4e40ce610 con 0x7ff4f01018f0 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdfea0 0x565260ce2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdfea0 0x565260ce2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48760/0 (socket says 172.21.15.77:48760) 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 -- 172.21.15.77:0/3439499158 learned_addr learned my addr 172.21.15.77:0/3439499158 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff4d80a28a0 0x7ff4d80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 -- 172.21.15.77:0/3439499158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 msgr2=0x565260cdf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 --2- 172.21.15.77:0/3439499158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260cdf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 -- 172.21.15.77:0/3439499158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cda340 msgr2=0x565260cda760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T09:44:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 --2- 172.21.15.77:0/3439499158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cda340 0x565260cda760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 -- 172.21.15.77:0/3439499158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565260c00460 con 0x565260cdfea0 2024-03-16T09:44:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff4d80a28a0 0x7ff4d80a4d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8012b20 tx=0x7ff4e8009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.107+0000 7ff4f4b71640 1 --2- 172.21.15.77:0/3439499158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdfea0 0x565260ce2330 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800fd40 tx=0x7ff4e8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a489714664b44632 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.108+0000 7ff4d67fc640 1 -- 172.21.15.77:0/3439499158 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8022070 con 0x565260cdfea0 2024-03-16T09:44:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.108+0000 7ff4d67fc640 1 -- 172.21.15.77:0/3439499158 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4e8003e20 con 0x565260cdfea0 2024-03-16T09:44:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.108+0000 7ff4d67fc640 1 -- 172.21.15.77:0/3439499158 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e80043b0 con 0x565260cdfea0 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.109+0000 7ff4f91c1740 1 -- 172.21.15.77:0/3439499158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdfea0 msgr2=0x565260ce2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.109+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/3439499158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdfea0 0x565260ce2330 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800fd40 tx=0x7ff4e8019000 comp rx=0 tx=0).stop 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.109+0000 7ff4f91c1740 1 -- 172.21.15.77:0/3439499158 shutdown_connections 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.109+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/3439499158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdfea0 0x565260ce2330 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.109+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/3439499158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260cdf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.109+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/3439499158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cda340 0x565260cda760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.110+0000 7ff4f91c1740 1 -- 172.21.15.77:0/3439499158 >> 172.21.15.77:0/3439499158 conn(0x565260cd58b0 msgr2=0x565260cd5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.110+0000 7ff4f91c1740 1 -- 172.21.15.77:0/3439499158 shutdown_connections 2024-03-16T09:44:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.110+0000 7ff4f91c1740 1 -- 172.21.15.77:0/3439499158 wait complete. 2024-03-16T09:44:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.110+0000 7ff4f91c1740 1 -- start start 2024-03-16T09:44:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.110+0000 7ff4f91c1740 1 Processor -- start 2024-03-16T09:44:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cda340 0x565260ce4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 0x565260cebbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565260ce2df0 con 0x565260cda340 2024-03-16T09:44:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x565260bfbd70 con 0x565260cdfea0 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x565260ce2f70 con 0x565260cdad30 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f91c1740 10 client.0 _ll_register_callbacks cb 0x565260953a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cda340 0x565260ce4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cda340 0x565260ce4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48768/0 (socket says 172.21.15.77:48768) 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 -- 172.21.15.77:0/388476025 learned_addr learned my addr 172.21.15.77:0/388476025 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4ef7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4f4b71640 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 0x565260cebbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 -- 172.21.15.77:0/388476025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 msgr2=0x565260ce4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 -- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 msgr2=0x565260cebbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 0x565260cebbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.111+0000 7ff4effff640 1 -- 172.21.15.77:0/388476025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565260954210 con 0x565260cda340 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4effff640 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cda340 0x565260ce4660 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006010 tx=0x7ff4e0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c622b24d076c33d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/388476025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0044020 con 0x565260cda340 2024-03-16T09:44:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/388476025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff4e0005240 con 0x565260cda340 2024-03-16T09:44:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4e8003e20 con 0x565260cda340 2024-03-16T09:44:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/388476025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff4e0005d50 con 0x565260cda340 2024-03-16T09:44:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4f91c1740 10 client.16050 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x565260a20620 con 0x565260cda340 2024-03-16T09:44:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/388476025 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4e8003e20 con 0x565260cda340 2024-03-16T09:44:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/388476025 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff4e00038f0 con 0x565260cda340 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4f91c1740 10 client.16050 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4f91c1740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4f91c1740 20 client.16050 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.112+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260cf04e0 0x565260cf28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.113+0000 7ff4f91c1740 4 client.16050 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.113+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565260c4cf80 con 0x565260cf04e0 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.113+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260cf04e0 0x565260cf28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.113+0000 7ff4cf7fe640 10 client.16050 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.113+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260cf04e0 0x565260cf28d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4325c0d3ba97431c server_cookie=f37036b64a730019 in_seq=0 out_seq=0 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.114+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/388476025 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565260c4cf80 con 0x565260cf04e0 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.114+0000 7ff4cf7fe640 10 client.16050 handle_command_reply: tid=0 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.114+0000 7ff4f91c1740 1 client.16050 shutdown 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cda340 msgr2=0x565260ce4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cda340 0x565260ce4660 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006010 tx=0x7ff4e0004270 comp rx=0 tx=0).stop 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 shutdown_connections 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260cf04e0 0x565260cf28d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 0x565260cebbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/388476025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cda340 0x565260ce4660 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 >> 172.21.15.77:0/388476025 conn(0x565260cd58b0 msgr2=0x565260cd4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4cf7fe640 0 client.16050 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 shutdown_connections 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 1 -- 172.21.15.77:0/388476025 wait complete. 2024-03-16T09:44:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.115+0000 7ff4f91c1740 20 client.16050 trim_cache size 0 max 16384 2024-03-16T09:44:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 Processor -- start 2024-03-16T09:44:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 -- start start 2024-03-16T09:44:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdad30 0x565260cd4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdfea0 0x565260cebba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cd60f0 0x565260cd4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01ad680 con 0x565260cdad30 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01aaf50 con 0x565260cd60f0 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01a8da0 con 0x565260cdfea0 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdad30 0x565260cd4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.134+0000 7ff4effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdad30 0x565260cd4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48774/0 (socket says 172.21.15.77:48774) 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4effff640 1 -- 172.21.15.77:0/2220944674 learned_addr learned my addr 172.21.15.77:0/2220944674 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdfea0 0x565260cebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4effff640 1 -- 172.21.15.77:0/2220944674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdfea0 msgr2=0x565260cebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4effff640 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdfea0 0x565260cebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4effff640 1 -- 172.21.15.77:0/2220944674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cd60f0 msgr2=0x565260cd4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4effff640 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cd60f0 0x565260cd4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4effff640 1 -- 172.21.15.77:0/2220944674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565260cdb520 con 0x565260cdad30 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4effff640 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdad30 0x565260cd4320 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0003d90 tx=0x7ff4e0003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152232a11bcb5cde server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4d5ffb640 1 -- 172.21.15.77:0/2220944674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0044070 con 0x565260cdad30 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4d5ffb640 1 -- 172.21.15.77:0/2220944674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4e0003aa0 con 0x565260cdad30 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.135+0000 7ff4d5ffb640 1 -- 172.21.15.77:0/2220944674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0040c80 con 0x565260cdad30 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.136+0000 7ff4f91c1740 1 -- 172.21.15.77:0/2220944674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdad30 msgr2=0x565260cd4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.136+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdad30 0x565260cd4320 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0003d90 tx=0x7ff4e0003f80 comp rx=0 tx=0).stop 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.136+0000 7ff4f91c1740 1 -- 172.21.15.77:0/2220944674 shutdown_connections 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.136+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cd60f0 0x565260cd4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.136+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdfea0 0x565260cebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.136+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/2220944674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cdad30 0x565260cd4320 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.136+0000 7ff4f91c1740 1 -- 172.21.15.77:0/2220944674 >> 172.21.15.77:0/2220944674 conn(0x565260cd58b0 msgr2=0x565260cd8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.137+0000 7ff4f91c1740 1 -- 172.21.15.77:0/2220944674 shutdown_connections 2024-03-16T09:44:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.137+0000 7ff4f91c1740 1 -- 172.21.15.77:0/2220944674 wait complete. 2024-03-16T09:44:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.137+0000 7ff4f91c1740 1 -- start start 2024-03-16T09:44:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.137+0000 7ff4f91c1740 1 Processor -- start 2024-03-16T09:44:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cd60f0 0x565260ce8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4f91c1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 0x565260cea440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01a9410 con 0x565260cd60f0 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01aa880 con 0x565260cdfea0 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4f91c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01a9ae0 con 0x565260cdad30 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cd60f0 0x565260ce8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4f91c1740 10 client.0 _ll_register_callbacks cb 0x7ff4e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4effff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cd60f0 0x565260ce8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48786/0 (socket says 172.21.15.77:48786) 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4effff640 1 -- 172.21.15.77:0/996222804 learned_addr learned my addr 172.21.15.77:0/996222804 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4effff640 1 -- 172.21.15.77:0/996222804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 msgr2=0x565260ce9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4effff640 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4effff640 1 -- 172.21.15.77:0/996222804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 msgr2=0x565260cea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.138+0000 7ff4effff640 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 0x565260cea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4effff640 1 -- 172.21.15.77:0/996222804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565260ceaa30 con 0x565260cd60f0 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4effff640 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cd60f0 0x565260ce8f40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0003c90 tx=0x7ff4e00098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403eb1f6a731dd40 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/996222804 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0044070 con 0x565260cd60f0 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/996222804 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff4e0040a00 con 0x565260cd60f0 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4e0003aa0 con 0x565260cd60f0 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/996222804 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff4e0040c00 con 0x565260cd60f0 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4f91c1740 10 client.16056 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.139+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x565260cebd40 con 0x565260cd60f0 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/996222804 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4e0003aa0 con 0x565260cd60f0 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/996222804 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff4e0065020 con 0x565260cd60f0 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4f91c1740 10 client.16056 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4f91c1740 10 client.16056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4f91c1740 20 client.16056 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260d5f600 0x565260d61a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4f91c1740 4 client.16056 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565260c472d0 con 0x565260d5f600 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.140+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260d5f600 0x565260d61a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.141+0000 7ff4ef7fe640 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260d5f600 0x565260d61a20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=496950829ca55f1f server_cookie=c1ca5a7d10cf7667 in_seq=0 out_seq=0 2024-03-16T09:44:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.141+0000 7ff4cf7fe640 10 client.16056 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.176+0000 7ff4cf7fe640 1 -- 172.21.15.77:0/996222804 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565260c472d0 con 0x565260d5f600 2024-03-16T09:44:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.176+0000 7ff4cf7fe640 10 client.16056 handle_command_reply: tid=0 2024-03-16T09:44:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.176+0000 7ff4f91c1740 1 client.16056 shutdown 2024-03-16T09:44:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cd60f0 msgr2=0x565260ce8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cd60f0 0x565260ce8f40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0003c90 tx=0x7ff4e00098a0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 shutdown_connections 2024-03-16T09:44:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x565260d5f600 0x565260d61a20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565260cdfea0 0x565260cea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4cf7fe640 0 client.16056 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565260cdad30 0x565260ce9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 --2- 172.21.15.77:0/996222804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565260cd60f0 0x565260ce8f40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 >> 172.21.15.77:0/996222804 conn(0x565260cd58b0 msgr2=0x565260cd4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 shutdown_connections 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 1 -- 172.21.15.77:0/996222804 wait complete. 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.177+0000 7ff4f91c1740 20 client.16056 trim_cache size 0 max 16384 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:44:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:44:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff4d80a28a0 msgr2=0x7ff4d80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff4d80a28a0 0x7ff4d80a4d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8012b20 tx=0x7ff4e8009920 comp rx=0 tx=0).stop 2024-03-16T09:44:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01018f0 msgr2=0x7ff4f019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01018f0 0x7ff4f019a1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b450 tx=0x7ff4e400b910 comp rx=0 tx=0).stop 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 shutdown_connections 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff4d80a28a0 0x7ff4d80a4d60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4f01022b0 0x7ff4f01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4f01018f0 0x7ff4f019a1d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 --2- 172.21.15.77:0/554079653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f0100700 0x7ff4f0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 >> 172.21.15.77:0/554079653 conn(0x7ff4f00fbc70 msgr2=0x7ff4f00fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 shutdown_connections 2024-03-16T09:44:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:20.178+0000 7ff4f65d4640 1 -- 172.21.15.77:0/554079653 wait complete. 2024-03-16T09:44:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:20 smithi077 ceph-mon[32171]: pgmap v419: 97 pgs: 97 active+clean; 6.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 550 MiB/s wr, 165 op/s 2024-03-16T09:44:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:20 smithi129 ceph-mon[35841]: pgmap v419: 97 pgs: 97 active+clean; 6.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 550 MiB/s wr, 165 op/s 2024-03-16T09:44:21.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:20 smithi119 ceph-mon[35632]: pgmap v419: 97 pgs: 97 active+clean; 6.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 550 MiB/s wr, 165 op/s 2024-03-16T09:44:22.244 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:22.244 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 initial writers = 709153.00 kB/sec 2024-03-16T09:44:22.244 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 initial writers = 709143.96 kB/sec 2024-03-16T09:44:22.244 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 709153.00 kB/sec 2024-03-16T09:44:22.244 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 709153.00 kB/sec 2024-03-16T09:44:22.244 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 709153.00 kB/sec 2024-03-16T09:44:22.244 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 10485760.00 kB 2024-03-16T09:44:22.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:21 smithi077 ceph-mon[32171]: pgmap v420: 97 pgs: 97 active+clean; 8.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 672 MiB/s wr, 193 op/s 2024-03-16T09:44:22.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:21 smithi129 ceph-mon[35841]: pgmap v420: 97 pgs: 97 active+clean; 8.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 672 MiB/s wr, 193 op/s 2024-03-16T09:44:22.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:21 smithi119 ceph-mon[35632]: pgmap v420: 97 pgs: 97 active+clean; 8.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 672 MiB/s wr, 193 op/s 2024-03-16T09:44:25.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:24 smithi119 ceph-mon[35632]: pgmap v421: 97 pgs: 97 active+clean; 9.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 651 MiB/s wr, 172 op/s 2024-03-16T09:44:25.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:24 smithi077 ceph-mon[32171]: pgmap v421: 97 pgs: 97 active+clean; 9.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 651 MiB/s wr, 172 op/s 2024-03-16T09:44:25.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:24 smithi129 ceph-mon[35841]: pgmap v421: 97 pgs: 97 active+clean; 9.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 651 MiB/s wr, 172 op/s 2024-03-16T09:44:27.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:26 smithi119 ceph-mon[35632]: pgmap v422: 97 pgs: 97 active+clean; 9.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 622 MiB/s wr, 166 op/s 2024-03-16T09:44:27.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:26 smithi077 ceph-mon[32171]: pgmap v422: 97 pgs: 97 active+clean; 9.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 622 MiB/s wr, 166 op/s 2024-03-16T09:44:27.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:26 smithi129 ceph-mon[35841]: pgmap v422: 97 pgs: 97 active+clean; 9.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 622 MiB/s wr, 166 op/s 2024-03-16T09:44:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:28 smithi077 ceph-mon[32171]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 565 MiB/s wr, 150 op/s 2024-03-16T09:44:29.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:28 smithi129 ceph-mon[35841]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 565 MiB/s wr, 150 op/s 2024-03-16T09:44:29.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:28 smithi119 ceph-mon[35632]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 565 MiB/s wr, 150 op/s 2024-03-16T09:44:30.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:30 smithi129 ceph-mon[35841]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 417 MiB/s wr, 113 op/s 2024-03-16T09:44:30.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:30 smithi119 ceph-mon[35632]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 417 MiB/s wr, 113 op/s 2024-03-16T09:44:30.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:30 smithi077 ceph-mon[32171]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 417 MiB/s wr, 113 op/s 2024-03-16T09:44:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:32 smithi077 ceph-mon[32171]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 508 MiB/s wr, 137 op/s 2024-03-16T09:44:33.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:32 smithi129 ceph-mon[35841]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 508 MiB/s wr, 137 op/s 2024-03-16T09:44:33.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 -- 172.21.15.77:0/2864697982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a90101930 msgr2=0x7f3a90101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2864697982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a90101930 0x7f3a90101db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009900 tx=0x7f3a8002bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:33.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 -- 172.21.15.77:0/2864697982 shutdown_connections 2024-03-16T09:44:33.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2864697982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a901022f0 0x7f3a90108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2864697982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a90101930 0x7f3a90101db0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2864697982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a90100740 0x7f3a90100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 -- 172.21.15.77:0/2864697982 >> 172.21.15.77:0/2864697982 conn(0x7f3a900fbcb0 msgr2=0x7f3a900fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:33.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.317+0000 7f3a97d97640 1 -- 172.21.15.77:0/2864697982 shutdown_connections 2024-03-16T09:44:33.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.318+0000 7f3a97d97640 1 -- 172.21.15.77:0/2864697982 wait complete. 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.318+0000 7f3a97d97640 1 Processor -- start 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a97d97640 1 -- start start 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a97d97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a90100740 0x7f3a90195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a97d97640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a90101930 0x7f3a90195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a97d97640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a901022f0 0x7f3a9019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a97d97640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9010b1c0 con 0x7f3a90100740 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a97d97640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9010b040 con 0x7f3a901022f0 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a97d97640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9010b340 con 0x7f3a90101930 2024-03-16T09:44:33.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a95b33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a90100740 0x7f3a90195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a95b33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a90100740 0x7f3a90195720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52068/0 (socket says 172.21.15.77:52068) 2024-03-16T09:44:33.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.319+0000 7f3a95b33640 1 -- 172.21.15.77:0/2987867545 learned_addr learned my addr 172.21.15.77:0/2987867545 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:33.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a95b33640 1 -- 172.21.15.77:0/2987867545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a90101930 msgr2=0x7f3a90195c60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a95b33640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a90101930 0x7f3a90195c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a95b33640 1 -- 172.21.15.77:0/2987867545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a901022f0 msgr2=0x7f3a9019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a95b33640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a901022f0 0x7f3a9019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a95b33640 1 -- 172.21.15.77:0/2987867545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9019d340 con 0x7f3a90100740 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a95b33640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a90100740 0x7f3a90195720 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00e9c0 tx=0x7f3a8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a86ffd640 1 -- 172.21.15.77:0/2987867545 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00cd80 con 0x7f3a90100740 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a86ffd640 1 -- 172.21.15.77:0/2987867545 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a8c0035f0 con 0x7f3a90100740 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.320+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9019d630 con 0x7f3a90100740 2024-03-16T09:44:33.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.321+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9019db80 con 0x7f3a90100740 2024-03-16T09:44:33.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.321+0000 7f3a86ffd640 1 -- 172.21.15.77:0/2987867545 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c010690 con 0x7f3a90100740 2024-03-16T09:44:33.327 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.324+0000 7f3a86ffd640 1 -- 172.21.15.77:0/2987867545 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3a8c0037a0 con 0x7f3a90100740 2024-03-16T09:44:33.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.325+0000 7f3a9a986740 1 Processor -- start 2024-03-16T09:44:33.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.325+0000 7f3a9a986740 1 -- start start 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.325+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c4b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.325+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c50830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.325+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c53330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.325+0000 7f3a9a986740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599b9bb0a40 con 0x5599b9c50ea0 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a9a986740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5599b9b59920 con 0x5599b9c4bd30 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a9a986740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5599b9bacae0 con 0x5599b9c4b340 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c53330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c53330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52082/0 (socket says 172.21.15.77:52082) 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 -- 172.21.15.77:0/3309603428 learned_addr learned my addr 172.21.15.77:0/3309603428 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a96334640 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c4b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95b33640 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c50830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 -- 172.21.15.77:0/3309603428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 msgr2=0x5599b9c4b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c4b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 -- 172.21.15.77:0/3309603428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 msgr2=0x5599b9c50830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c50830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.326+0000 7f3a95332640 1 -- 172.21.15.77:0/3309603428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b9b71460 con 0x5599b9c50ea0 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.327+0000 7f3a86ffd640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a7c0a4b30 0x7f3a7c0a6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.327+0000 7f3a95332640 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c53330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3a80000c00 tx=0x7f3a80005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9e9e315cce725c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.327+0000 7f3a96334640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a7c0a4b30 0x7f3a7c0a6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.327+0000 7f3a86ffd640 1 -- 172.21.15.77:0/2987867545 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3a8c0cf010 con 0x7f3a90100740 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.327+0000 7f3a7bfff640 1 -- 172.21.15.77:0/3309603428 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80042020 con 0x5599b9c50ea0 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.327+0000 7f3a7bfff640 1 -- 172.21.15.77:0/3309603428 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a80009520 con 0x5599b9c50ea0 2024-03-16T09:44:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.327+0000 7f3a7bfff640 1 -- 172.21.15.77:0/3309603428 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80003d00 con 0x5599b9c50ea0 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a9a986740 1 -- 172.21.15.77:0/3309603428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 msgr2=0x5599b9c53330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a96334640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a7c0a4b30 0x7f3a7c0a6ff0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5599b9c4c590 tx=0x7f3a88007520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a9a986740 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c53330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3a80000c00 tx=0x7f3a80005760 comp rx=0 tx=0).stop 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a9a986740 1 -- 172.21.15.77:0/3309603428 shutdown_connections 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a9a986740 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c53330 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a9a986740 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c50830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a9a986740 1 --2- 172.21.15.77:0/3309603428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c4b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.328+0000 7f3a9a986740 1 -- 172.21.15.77:0/3309603428 >> 172.21.15.77:0/3309603428 conn(0x5599b9c468b0 msgr2=0x5599b9c46cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.329+0000 7f3a9a986740 1 -- 172.21.15.77:0/3309603428 shutdown_connections 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.329+0000 7f3a9a986740 1 -- 172.21.15.77:0/3309603428 wait complete. 2024-03-16T09:44:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.329+0000 7f3a9a986740 1 -- start start 2024-03-16T09:44:33.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.329+0000 7f3a9a986740 1 Processor -- start 2024-03-16T09:44:33.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4b340 0x5599b9c555f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4bd30 0x5599b9c55b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9c5cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a9a986740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599b9c53df0 con 0x5599b9c4bd30 2024-03-16T09:44:33.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a9a986740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5599b9b6cd70 con 0x5599b9c4b340 2024-03-16T09:44:33.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a9a986740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5599b9c53f70 con 0x5599b9c50ea0 2024-03-16T09:44:33.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a96334640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9c5cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a95332640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4bd30 0x5599b9c55b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a96334640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9c5cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:39414/0 (socket says 172.21.15.77:39414) 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a96334640 1 -- 172.21.15.77:0/515541915 learned_addr learned my addr 172.21.15.77:0/515541915 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a9a986740 10 client.0 _ll_register_callbacks cb 0x5599b98c4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.330+0000 7f3a95b33640 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4b340 0x5599b9c555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a95332640 1 -- 172.21.15.77:0/515541915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 msgr2=0x5599b9c5cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a95332640 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9c5cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a95332640 1 -- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4b340 msgr2=0x5599b9c555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a95332640 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4b340 0x5599b9c555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a95332640 1 -- 172.21.15.77:0/515541915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b98c5210 con 0x5599b9c4bd30 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a95332640 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4bd30 0x5599b9c55b30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x5599b9c47820 tx=0x7f3a80005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b00027dcf5d5bf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a797fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a797fa640 1 -- 172.21.15.77:0/515541915 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80042030 con 0x5599b9c4bd30 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a80009520 con 0x5599b9c4bd30 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.331+0000 7f3a797fa640 1 -- 172.21.15.77:0/515541915 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3a80034850 con 0x5599b9c4bd30 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.332+0000 7f3a797fa640 1 -- 172.21.15.77:0/515541915 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3a80034a50 con 0x5599b9c4bd30 2024-03-16T09:44:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.332+0000 7f3a9a986740 10 client.16065 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:33.337 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.332+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5599b9991620 con 0x5599b9c4bd30 2024-03-16T09:44:33.337 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.332+0000 7f3a797fa640 1 -- 172.21.15.77:0/515541915 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a80009520 con 0x5599b9c4bd30 2024-03-16T09:44:33.337 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a797fa640 1 -- 172.21.15.77:0/515541915 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3a80003a80 con 0x5599b9c4bd30 2024-03-16T09:44:33.337 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a9a986740 10 client.16065 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a9a986740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a9a986740 20 client.16065 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a9a986740 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9c61040 0x5599b9c63430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a9a986740 4 client.16065 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599b9bbdf80 con 0x5599b9c61040 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.333+0000 7f3a95b33640 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9c61040 0x5599b9c63430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.336+0000 7f3a797fa640 10 client.16065 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:33.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.336+0000 7f3a95b33640 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9c61040 0x5599b9c63430 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=939cb53cdc0cfd97 server_cookie=feb1e90aa9b1d592 in_seq=0 out_seq=0 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a797fa640 1 -- 172.21.15.77:0/515541915 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599b9bbdf80 con 0x5599b9c61040 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a797fa640 10 client.16065 handle_command_reply: tid=0 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 client.16065 shutdown 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4bd30 msgr2=0x5599b9c55b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4bd30 0x5599b9c55b30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x5599b9c47820 tx=0x7f3a80005850 comp rx=0 tx=0).stop 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 shutdown_connections 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9c61040 0x5599b9c63430 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9c5cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4bd30 0x5599b9c55b30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 --2- 172.21.15.77:0/515541915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4b340 0x5599b9c555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 >> 172.21.15.77:0/515541915 conn(0x5599b9c468b0 msgr2=0x5599b9c45bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a797fa640 0 client.16065 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 shutdown_connections 2024-03-16T09:44:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.337+0000 7f3a9a986740 1 -- 172.21.15.77:0/515541915 wait complete. 2024-03-16T09:44:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.338+0000 7f3a9a986740 20 client.16065 trim_cache size 0 max 16384 2024-03-16T09:44:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.357+0000 7f3a9a986740 1 Processor -- start 2024-03-16T09:44:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.357+0000 7f3a9a986740 1 -- start start 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c45a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c68c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a9a986740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901ad550 con 0x5599b9c50ea0 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a9a986740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901aae20 con 0x5599b9c4bd30 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a9a986740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901a8c70 con 0x5599b9c4b340 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c68c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c68c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52100/0 (socket says 172.21.15.77:52100) 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 -- 172.21.15.77:0/547581501 learned_addr learned my addr 172.21.15.77:0/547581501 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a95b33640 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c45a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 -- 172.21.15.77:0/547581501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 msgr2=0x5599b9c45a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c45a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 -- 172.21.15.77:0/547581501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 msgr2=0x5599b9c5cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 -- 172.21.15.77:0/547581501 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b9c47620 con 0x5599b9c50ea0 2024-03-16T09:44:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.358+0000 7f3a96334640 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c68c10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3a8801e750 tx=0x7f3a88012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecef5d2cfa0766f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.359+0000 7f3a7b7fe640 1 -- 172.21.15.77:0/547581501 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8802b070 con 0x5599b9c50ea0 2024-03-16T09:44:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.359+0000 7f3a7b7fe640 1 -- 172.21.15.77:0/547581501 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a88010df0 con 0x5599b9c50ea0 2024-03-16T09:44:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.359+0000 7f3a7b7fe640 1 -- 172.21.15.77:0/547581501 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880193a0 con 0x5599b9c50ea0 2024-03-16T09:44:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.360+0000 7f3a9a986740 1 -- 172.21.15.77:0/547581501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 msgr2=0x5599b9c68c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.360+0000 7f3a9a986740 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c68c10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3a8801e750 tx=0x7f3a88012000 comp rx=0 tx=0).stop 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.360+0000 7f3a9a986740 1 -- 172.21.15.77:0/547581501 shutdown_connections 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.360+0000 7f3a9a986740 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c50ea0 0x5599b9c68c10 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.360+0000 7f3a9a986740 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.360+0000 7f3a9a986740 1 --2- 172.21.15.77:0/547581501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c4b340 0x5599b9c45a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.360+0000 7f3a9a986740 1 -- 172.21.15.77:0/547581501 >> 172.21.15.77:0/547581501 conn(0x5599b9c468b0 msgr2=0x5599b9c45320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.361+0000 7f3a9a986740 1 -- 172.21.15.77:0/547581501 shutdown_connections 2024-03-16T09:44:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.361+0000 7f3a9a986740 1 -- 172.21.15.77:0/547581501 wait complete. 2024-03-16T09:44:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.361+0000 7f3a9a986740 1 -- start start 2024-03-16T09:44:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.361+0000 7f3a9a986740 1 Processor -- start 2024-03-16T09:44:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4b340 0x5599b9c5bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a9a986740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9cc7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a9a986740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901a92e0 con 0x5599b9c4b340 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a9a986740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901aa750 con 0x5599b9c4bd30 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a9a986740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901a99b0 con 0x5599b9c50ea0 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a9a986740 10 client.0 _ll_register_callbacks cb 0x7f3a8c0d29c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a95b33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4b340 0x5599b9c5bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a95332640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a95b33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4b340 0x5599b9c5bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52106/0 (socket says 172.21.15.77:52106) 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a95332640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:39988/0 (socket says 172.21.15.77:39988) 2024-03-16T09:44:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.362+0000 7f3a95b33640 1 -- 172.21.15.77:0/1959944767 learned_addr learned my addr 172.21.15.77:0/1959944767 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.363+0000 7f3a95b33640 1 -- 172.21.15.77:0/1959944767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 msgr2=0x5599b9cc7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.363+0000 7f3a95b33640 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9cc7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.363+0000 7f3a95b33640 1 -- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 msgr2=0x5599b9c5c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.363+0000 7f3a95b33640 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.363+0000 7f3a95b33640 1 -- 172.21.15.77:0/1959944767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b9cc7780 con 0x5599b9c4b340 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.363+0000 7f3a95332640 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.363+0000 7f3a95b33640 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4b340 0x5599b9c5bae0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00cfd0 tx=0x7f3a8c005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29bb6aacf7f12d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.364+0000 7f3a797fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.364+0000 7f3a797fa640 1 -- 172.21.15.77:0/1959944767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c09c0c0 con 0x5599b9c4b340 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.364+0000 7f3a797fa640 1 -- 172.21.15.77:0/1959944767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3a8c099cc0 con 0x5599b9c4b340 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.364+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a88010df0 con 0x5599b9c4b340 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.364+0000 7f3a797fa640 1 -- 172.21.15.77:0/1959944767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3a8c09b9c0 con 0x5599b9c4b340 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 10 client.16071 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5599b9cc8a80 con 0x5599b9c4b340 2024-03-16T09:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a797fa640 1 -- 172.21.15.77:0/1959944767 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a88010df0 con 0x5599b9c4b340 2024-03-16T09:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a797fa640 1 -- 172.21.15.77:0/1959944767 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3a8c0d10c0 con 0x5599b9c4b340 2024-03-16T09:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 10 client.16071 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 20 client.16071 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9cc93b0 0x5599b9ccb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 4 client.16071 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.365+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5599b9bb82d0 con 0x5599b9cc93b0 2024-03-16T09:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.366+0000 7f3a95332640 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9cc93b0 0x5599b9ccb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:33.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.369+0000 7f3a797fa640 10 client.16071 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:33.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.369+0000 7f3a95332640 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9cc93b0 0x5599b9ccb7d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ac257092eb868f1 server_cookie=78a40e10d2a087e8 in_seq=0 out_seq=0 2024-03-16T09:44:33.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:32 smithi119 ceph-mon[35632]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 508 MiB/s wr, 137 op/s 2024-03-16T09:44:33.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.447+0000 7f3a797fa640 1 -- 172.21.15.77:0/1959944767 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5599b9bb82d0 con 0x5599b9cc93b0 2024-03-16T09:44:33.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.447+0000 7f3a797fa640 10 client.16071 handle_command_reply: tid=0 2024-03-16T09:44:33.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.447+0000 7f3a9a986740 1 client.16071 shutdown 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4b340 msgr2=0x5599b9c5bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4b340 0x5599b9c5bae0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00cfd0 tx=0x7f3a8c005cb0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 shutdown_connections 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x5599b9cc93b0 0x5599b9ccb7d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5599b9c50ea0 0x5599b9cc7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599b9c4bd30 0x5599b9c5c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a797fa640 0 client.16071 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 --2- 172.21.15.77:0/1959944767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599b9c4b340 0x5599b9c5bae0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 >> 172.21.15.77:0/1959944767 conn(0x5599b9c468b0 msgr2=0x5599b9c45bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 shutdown_connections 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 1 -- 172.21.15.77:0/1959944767 wait complete. 2024-03-16T09:44:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.448+0000 7f3a9a986740 20 client.16071 trim_cache size 0 max 16384 2024-03-16T09:44:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:44:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:44:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:44:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:44:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a7c0a4b30 msgr2=0x7f3a7c0a6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a7c0a4b30 0x7f3a7c0a6ff0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5599b9c4c590 tx=0x7f3a88007520 comp rx=0 tx=0).stop 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a90100740 msgr2=0x7f3a90195720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a90100740 0x7f3a90195720 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00e9c0 tx=0x7f3a8c00ee80 comp rx=0 tx=0).stop 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 shutdown_connections 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a7c0a4b30 0x7f3a7c0a6ff0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a901022f0 0x7f3a9019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a90101930 0x7f3a90195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 --2- 172.21.15.77:0/2987867545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a90100740 0x7f3a90195720 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 >> 172.21.15.77:0/2987867545 conn(0x7f3a900fbcb0 msgr2=0x7f3a90104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.449+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 shutdown_connections 2024-03-16T09:44:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:33.450+0000 7f3a97d97640 1 -- 172.21.15.77:0/2987867545 wait complete. 2024-03-16T09:44:34.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:34 smithi077 ceph-mon[32171]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 456 MiB/s wr, 121 op/s 2024-03-16T09:44:34.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:34 smithi129 ceph-mon[35841]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 456 MiB/s wr, 121 op/s 2024-03-16T09:44:34.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:34 smithi119 ceph-mon[35632]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 456 MiB/s wr, 121 op/s 2024-03-16T09:44:35.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:44:35.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:44:35.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.425+0000 7f7d01675640 1 -- 172.21.15.77:0/1861451577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc102360 msgr2=0x7f7cfc104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.425+0000 7f7d01675640 1 --2- 172.21.15.77:0/1861451577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc102360 0x7f7cfc104750 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009900 tx=0x7f7cec02bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.425+0000 7f7d01675640 1 -- 172.21.15.77:0/1861451577 shutdown_connections 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.425+0000 7f7d01675640 1 --2- 172.21.15.77:0/1861451577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cfc1075c0 0x7f7cfc1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.425+0000 7f7d01675640 1 --2- 172.21.15.77:0/1861451577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7cfc104c90 0x7f7cfc107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.425+0000 7f7d01675640 1 --2- 172.21.15.77:0/1861451577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc102360 0x7f7cfc104750 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.425+0000 7f7d01675640 1 -- 172.21.15.77:0/1861451577 >> 172.21.15.77:0/1861451577 conn(0x7f7cfc0fbc90 msgr2=0x7f7cfc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.426+0000 7f7d01675640 1 -- 172.21.15.77:0/1861451577 shutdown_connections 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.426+0000 7f7d01675640 1 -- 172.21.15.77:0/1861451577 wait complete. 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 Processor -- start 2024-03-16T09:44:35.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 -- start start 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc104c90 0x7f7cfc19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cfc1075c0 0x7f7cfc19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7cfc1a38b0 0x7f7cfc1a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc101c80 con 0x7f7cfc104c90 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc101b00 con 0x7f7cfc1a38b0 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7d01675640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc101e00 con 0x7f7cfc1075c0 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7cfaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc104c90 0x7f7cfc19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7cfaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc104c90 0x7f7cfc19e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41572/0 (socket says 172.21.15.77:41572) 2024-03-16T09:44:35.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.427+0000 7f7cfaffd640 1 -- 172.21.15.77:0/548591604 learned_addr learned my addr 172.21.15.77:0/548591604 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.428+0000 7f7cfb7fe640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7cfc1a38b0 0x7f7cfc1a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.428+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cfc1075c0 0x7f7cfc19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.428+0000 7f7cfaffd640 1 -- 172.21.15.77:0/548591604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cfc1075c0 msgr2=0x7f7cfc19e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.428+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cfc1075c0 0x7f7cfc19e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.428+0000 7f7cfaffd640 1 -- 172.21.15.77:0/548591604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7cfc1a38b0 msgr2=0x7f7cfc1a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.428+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7cfc1a38b0 0x7f7cfc1a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.429+0000 7f7cfaffd640 1 -- 172.21.15.77:0/548591604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cfc1a6310 con 0x7f7cfc104c90 2024-03-16T09:44:35.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.429+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc104c90 0x7f7cfc19e310 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7cec02c240 tx=0x7f7cec005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.429+0000 7f7cf8ff9640 1 -- 172.21.15.77:0/548591604 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec043020 con 0x7f7cfc104c90 2024-03-16T09:44:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.429+0000 7f7cf8ff9640 1 -- 172.21.15.77:0/548591604 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cec0043d0 con 0x7f7cfc104c90 2024-03-16T09:44:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.429+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cfc1a65a0 con 0x7f7cfc104c90 2024-03-16T09:44:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.429+0000 7f7cf8ff9640 1 -- 172.21.15.77:0/548591604 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec003a80 con 0x7f7cfc104c90 2024-03-16T09:44:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.429+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cfc1a6aa0 con 0x7f7cfc104c90 2024-03-16T09:44:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 Processor -- start 2024-03-16T09:44:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 -- start start 2024-03-16T09:44:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1170 0x555cf0df1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 0x555cf0df6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0df9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555cf0d56500 con 0x555cf0df6cd0 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555cf0cfebe0 con 0x555cf0df1b60 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7d04259740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555cf0d525a0 con 0x555cf0df1170 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7cf8ff9640 1 -- 172.21.15.77:0/548591604 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7cec046020 con 0x7f7cfc104c90 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7cf3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0df9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7cf3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0df9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41582/0 (socket says 172.21.15.77:41582) 2024-03-16T09:44:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.433+0000 7f7cf3fff640 1 -- 172.21.15.77:0/1620397705 learned_addr learned my addr 172.21.15.77:0/1620397705 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.434+0000 7f7cf3fff640 1 -- 172.21.15.77:0/1620397705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1170 msgr2=0x555cf0df1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.434+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1620397705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1170 0x555cf0df1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.434+0000 7f7cf3fff640 1 -- 172.21.15.77:0/1620397705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 msgr2=0x555cf0df6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.434+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1620397705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 0x555cf0df6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.434+0000 7f7cf3fff640 1 -- 172.21.15.77:0/1620397705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cf0d124b0 con 0x555cf0df6cd0 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.434+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1620397705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0df9160 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009bf0 tx=0x7f7ce40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3cea703f94ccab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.434+0000 7f7cf8ff9640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ce80a2880 0x7f7ce80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.435+0000 7f7cf8ff9640 1 -- 172.21.15.77:0/548591604 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7cec0f0b20 con 0x7f7cfc104c90 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.435+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1620397705 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4004030 con 0x555cf0df6cd0 2024-03-16T09:44:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.435+0000 7f7cfb7fe640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ce80a2880 0x7f7ce80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.435+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1620397705 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ce40041d0 con 0x555cf0df6cd0 2024-03-16T09:44:35.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.435+0000 7f7cf17fa640 1 -- 172.21.15.77:0/1620397705 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce40126b0 con 0x555cf0df6cd0 2024-03-16T09:44:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 -- 172.21.15.77:0/1620397705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 msgr2=0x555cf0df9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7cfb7fe640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ce80a2880 0x7f7ce80a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x555cf0df23c0 tx=0x7f7cf4009ae0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 --2- 172.21.15.77:0/1620397705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0df9160 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009bf0 tx=0x7f7ce40117b0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 -- 172.21.15.77:0/1620397705 shutdown_connections 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 --2- 172.21.15.77:0/1620397705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0df9160 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 --2- 172.21.15.77:0/1620397705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 0x555cf0df6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 --2- 172.21.15.77:0/1620397705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1170 0x555cf0df1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 -- 172.21.15.77:0/1620397705 >> 172.21.15.77:0/1620397705 conn(0x555cf0dec6e0 msgr2=0x555cf0decae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 -- 172.21.15.77:0/1620397705 shutdown_connections 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 -- 172.21.15.77:0/1620397705 wait complete. 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.436+0000 7f7d04259740 1 -- start start 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 1 Processor -- start 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df1170 0x555cf0dfb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 0x555cf0dfbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df6cd0 0x555cf0e02c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555cf0df9c80 con 0x555cf0df1170 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555cf0d12160 con 0x555cf0df1b60 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555cf0df9e00 con 0x555cf0df6cd0 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df1170 0x555cf0dfb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7d04259740 10 client.0 _ll_register_callbacks cb 0x555cf0cf6150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df1170 0x555cf0dfb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41586/0 (socket says 172.21.15.77:41586) 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 -- 172.21.15.77:0/736861398 learned_addr learned my addr 172.21.15.77:0/736861398 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 0x555cf0dfbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 -- 172.21.15.77:0/736861398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df6cd0 msgr2=0x555cf0e02c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df6cd0 0x555cf0e02c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 -- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 msgr2=0x555cf0dfbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 0x555cf0dfbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.437+0000 7f7cfaffd640 1 -- 172.21.15.77:0/736861398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cf0b347d0 con 0x555cf0df1170 2024-03-16T09:44:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df1170 0x555cf0dfb710 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0bd460 tx=0x7f7cec0bd010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32a85c76cc33bac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7ce27fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7ce27fc640 1 -- 172.21.15.77:0/736861398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0bd4d0 con 0x555cf0df1170 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ce40041d0 con 0x555cf0df1170 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7ce27fc640 1 -- 172.21.15.77:0/736861398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7cec0bd730 con 0x555cf0df1170 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7ce27fc640 1 -- 172.21.15.77:0/736861398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7cec0c1620 con 0x555cf0df1170 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7d04259740 10 client.16083 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555cf0a6a3f0 con 0x555cf0df1170 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.438+0000 7f7ce27fc640 1 -- 172.21.15.77:0/736861398 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ce40041d0 con 0x555cf0df1170 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7ce27fc640 1 -- 172.21.15.77:0/736861398 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7cec0d60f0 con 0x555cf0df1170 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7d04259740 10 client.16083 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7d04259740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7d04259740 20 client.16083 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7d04259740 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e07400 0x555cf0e09870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7d04259740 4 client.16083 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:35.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cf0d63a40 con 0x555cf0e07400 2024-03-16T09:44:35.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.439+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e07400 0x555cf0e09870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.441+0000 7f7ce27fc640 10 client.16083 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.441+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e07400 0x555cf0e09870 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=608b997c57e67d1e server_cookie=379db08a44eca3e2 in_seq=0 out_seq=0 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.441+0000 7f7ce27fc640 1 -- 172.21.15.77:0/736861398 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555cf0d63a40 con 0x555cf0e07400 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.441+0000 7f7ce27fc640 10 client.16083 handle_command_reply: tid=0 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 client.16083 shutdown 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df1170 msgr2=0x555cf0dfb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df1170 0x555cf0dfb710 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0bd460 tx=0x7f7cec0bd010 comp rx=0 tx=0).stop 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 shutdown_connections 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e07400 0x555cf0e09870 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df6cd0 0x555cf0e02c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1b60 0x555cf0dfbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 --2- 172.21.15.77:0/736861398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df1170 0x555cf0dfb710 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 >> 172.21.15.77:0/736861398 conn(0x555cf0dec6e0 msgr2=0x555cf0deb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7ce27fc640 0 client.16083 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 shutdown_connections 2024-03-16T09:44:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 1 -- 172.21.15.77:0/736861398 wait complete. 2024-03-16T09:44:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.442+0000 7f7d04259740 20 client.16083 trim_cache size 0 max 16384 2024-03-16T09:44:35.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.473+0000 7f7d04259740 1 Processor -- start 2024-03-16T09:44:35.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.473+0000 7f7d04259740 1 -- start start 2024-03-16T09:44:35.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7d04259740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dfba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7d04259740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 0x555cf0e02c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7d04259740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0deb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7d04259740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc1b2020 con 0x555cf0df6cd0 2024-03-16T09:44:35.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7d04259740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc1af8f0 con 0x555cf0df1170 2024-03-16T09:44:35.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7d04259740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc1ad740 con 0x555cf0df1b60 2024-03-16T09:44:35.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0deb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7cfaffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dfba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0deb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41588/0 (socket says 172.21.15.77:41588) 2024-03-16T09:44:35.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7cfaffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dfba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:34474/0 (socket says 172.21.15.77:34474) 2024-03-16T09:44:35.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.474+0000 7f7cfb7fe640 1 -- 172.21.15.77:0/1535005815 learned_addr learned my addr 172.21.15.77:0/1535005815 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.475+0000 7f7cfaffd640 1 -- 172.21.15.77:0/1535005815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 msgr2=0x555cf0e02c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.475+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/1535005815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 0x555cf0e02c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.475+0000 7f7cfaffd640 1 -- 172.21.15.77:0/1535005815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 msgr2=0x555cf0deb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.475+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/1535005815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0deb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.475+0000 7f7cfaffd640 1 -- 172.21.15.77:0/1535005815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cf0a2a0f0 con 0x555cf0df1170 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.475+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/1535005815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dfba30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0b9e50 tx=0x7f7cec0bb9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58b80f02d4e44178 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.476+0000 7f7cf0ff9640 1 -- 172.21.15.77:0/1535005815 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0bce30 con 0x555cf0df1170 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.476+0000 7f7cf0ff9640 1 -- 172.21.15.77:0/1535005815 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cec0bd4d0 con 0x555cf0df1170 2024-03-16T09:44:35.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.476+0000 7f7cf0ff9640 1 -- 172.21.15.77:0/1535005815 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0eeaf0 con 0x555cf0df1170 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 -- 172.21.15.77:0/1535005815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 msgr2=0x555cf0dfba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 --2- 172.21.15.77:0/1535005815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dfba30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0b9e50 tx=0x7f7cec0bb9e0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 -- 172.21.15.77:0/1535005815 shutdown_connections 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 --2- 172.21.15.77:0/1535005815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0deb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 --2- 172.21.15.77:0/1535005815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 0x555cf0e02c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 --2- 172.21.15.77:0/1535005815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dfba30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 -- 172.21.15.77:0/1535005815 >> 172.21.15.77:0/1535005815 conn(0x555cf0dec6e0 msgr2=0x555cf0dfb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:35.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 -- 172.21.15.77:0/1535005815 shutdown_connections 2024-03-16T09:44:35.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.477+0000 7f7d04259740 1 -- 172.21.15.77:0/1535005815 wait complete. 2024-03-16T09:44:35.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.478+0000 7f7d04259740 1 -- start start 2024-03-16T09:44:35.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.478+0000 7f7d04259740 1 Processor -- start 2024-03-16T09:44:35.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.478+0000 7f7d04259740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dffe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.478+0000 7f7d04259740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 0x555cf0e00340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.478+0000 7f7d04259740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0e01350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.478+0000 7f7d04259740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc1addb0 con 0x555cf0df6cd0 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.478+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0e01350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0e01350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41594/0 (socket says 172.21.15.77:41594) 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 -- 172.21.15.77:0/1843123687 learned_addr learned my addr 172.21.15.77:0/1843123687 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc1af220 con 0x555cf0df1170 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc1ae480 con 0x555cf0df1b60 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dffe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cf3fff640 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 0x555cf0e00340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 -- 172.21.15.77:0/1843123687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 msgr2=0x555cf0e00340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 0x555cf0e00340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 -- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 msgr2=0x555cf0dffe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dffe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 -- 172.21.15.77:0/1843123687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cf0d1d9d0 con 0x555cf0df6cd0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7cfb7fe640 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0e01350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4012c10 tx=0x7f7cf40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3001cfc84dda5e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.479+0000 7f7ce27fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.480+0000 7f7d04259740 10 client.0 _ll_register_callbacks cb 0x7f7cec0b9130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.480+0000 7f7ce27fc640 1 -- 172.21.15.77:0/1843123687 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4029670 con 0x555cf0df6cd0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.480+0000 7f7ce27fc640 1 -- 172.21.15.77:0/1843123687 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7cf4020430 con 0x555cf0df6cd0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.480+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cec0bd4d0 con 0x555cf0df6cd0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.480+0000 7f7ce27fc640 1 -- 172.21.15.77:0/1843123687 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7cf4012e50 con 0x555cf0df6cd0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.480+0000 7f7d04259740 10 client.16089 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.480+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=21}) -- 0x555cf0e028b0 con 0x555cf0df6cd0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7ce27fc640 1 -- 172.21.15.77:0/1843123687 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7cf400f040 con 0x555cf0df6cd0 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7d04259740 10 client.16089 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7d04259740 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:35.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7d04259740 20 client.16089 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:35.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7d04259740 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e6d840 0x555cf0e6fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7d04259740 4 client.16089 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:44:35.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555cf0d5dd90 con 0x555cf0e6d840 2024-03-16T09:44:35.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.481+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e6d840 0x555cf0e6fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.482+0000 7f7ce27fc640 10 client.16089 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:35.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.482+0000 7f7cfaffd640 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e6d840 0x555cf0e6fcd0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdb8716eb95f66e2 server_cookie=c506f32e82767188 in_seq=0 out_seq=0 2024-03-16T09:44:35.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7ce27fc640 1 -- 172.21.15.77:0/1843123687 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555cf0d5dd90 con 0x555cf0e6d840 2024-03-16T09:44:35.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7ce27fc640 10 client.16089 handle_command_reply: tid=0 2024-03-16T09:44:35.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 client.16089 shutdown 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 msgr2=0x555cf0e01350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0e01350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4012c10 tx=0x7f7cf40117b0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 shutdown_connections 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x555cf0e6d840 0x555cf0e6fcd0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7ce27fc640 0 client.16089 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555cf0df6cd0 0x555cf0e01350 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555cf0df1b60 0x555cf0e00340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 --2- 172.21.15.77:0/1843123687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555cf0df1170 0x555cf0dffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 >> 172.21.15.77:0/1843123687 conn(0x555cf0dec6e0 msgr2=0x555cf0def940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 shutdown_connections 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 1 -- 172.21.15.77:0/1843123687 wait complete. 2024-03-16T09:44:35.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.483+0000 7f7d04259740 20 client.16089 trim_cache size 0 max 16384 2024-03-16T09:44:35.489 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:44:35.489 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T09:44:35.489 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T09:44:35.489 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ce80a2880 msgr2=0x7f7ce80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ce80a2880 0x7f7ce80a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x555cf0df23c0 tx=0x7f7cf4009ae0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc104c90 msgr2=0x7f7cfc19e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc104c90 0x7f7cfc19e310 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7cec02c240 tx=0x7f7cec005e80 comp rx=0 tx=0).stop 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 shutdown_connections 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7ce80a2880 0x7f7ce80a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7cfc1a38b0 0x7f7cfc1a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cfc1075c0 0x7f7cfc19e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 --2- 172.21.15.77:0/548591604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc104c90 0x7f7cfc19e310 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 >> 172.21.15.77:0/548591604 conn(0x7f7cfc0fbc90 msgr2=0x7f7cfc102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 shutdown_connections 2024-03-16T09:44:35.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.484+0000 7f7d01675640 1 -- 172.21.15.77:0/548591604 wait complete. 2024-03-16T09:44:35.507 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T09:44:35.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:44:35.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 -- 172.21.15.77:0/943579283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308107e20 msgr2=0x7f3308108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 --2- 172.21.15.77:0/943579283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308107e20 0x7f3308108240 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009900 tx=0x7f32f802bc60 comp rx=0 tx=0).stop 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 -- 172.21.15.77:0/943579283 shutdown_connections 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 --2- 172.21.15.77:0/943579283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3308109930 0x7f3308110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 --2- 172.21.15.77:0/943579283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3308108f70 0x7f33081093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 --2- 172.21.15.77:0/943579283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308107e20 0x7f3308108240 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 -- 172.21.15.77:0/943579283 >> 172.21.15.77:0/943579283 conn(0x7f3308076df0 msgr2=0x7f3308079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.667+0000 7f330db5b640 1 -- 172.21.15.77:0/943579283 shutdown_connections 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.668+0000 7f330db5b640 1 -- 172.21.15.77:0/943579283 wait complete. 2024-03-16T09:44:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.668+0000 7f330db5b640 1 Processor -- start 2024-03-16T09:44:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f330db5b640 1 -- start start 2024-03-16T09:44:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f330db5b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3308107e20 0x7f330819e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f330db5b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3308108f70 0x7f330819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f330db5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308109930 0x7f33081a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f330db5b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3308113a40 con 0x7f3308109930 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f330db5b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f33081138c0 con 0x7f3308107e20 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f330db5b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3308113bc0 con 0x7f3308108f70 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f3307fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308109930 0x7f33081a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f3306ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3308108f70 0x7f330819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f3307fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308109930 0x7f33081a5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41618/0 (socket says 172.21.15.77:41618) 2024-03-16T09:44:35.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.669+0000 7f3307fff640 1 -- 172.21.15.77:0/3466342495 learned_addr learned my addr 172.21.15.77:0/3466342495 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3307fff640 1 -- 172.21.15.77:0/3466342495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3308108f70 msgr2=0x7f330819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3307fff640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3308108f70 0x7f330819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3307fff640 1 -- 172.21.15.77:0/3466342495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3308107e20 msgr2=0x7f330819e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3307fff640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3308107e20 0x7f330819e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3307fff640 1 -- 172.21.15.77:0/3466342495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330806b330 con 0x7f3308109930 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3307fff640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308109930 0x7f33081a5660 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f330000cce0 tx=0x7f330000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3304ff9640 1 -- 172.21.15.77:0/3466342495 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330000f450 con 0x7f3308109930 2024-03-16T09:44:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3304ff9640 1 -- 172.21.15.77:0/3466342495 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f330000faa0 con 0x7f3308109930 2024-03-16T09:44:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330806b620 con 0x7f3308109930 2024-03-16T09:44:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.670+0000 7f3304ff9640 1 -- 172.21.15.77:0/3466342495 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300018b70 con 0x7f3308109930 2024-03-16T09:44:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.671+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330806bbf0 con 0x7f3308109930 2024-03-16T09:44:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.673+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f330810f130 con 0x7f3308109930 2024-03-16T09:44:35.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.675+0000 7f3304ff9640 1 -- 172.21.15.77:0/3466342495 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3300018d10 con 0x7f3308109930 2024-03-16T09:44:35.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.675+0000 7f3304ff9640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f32f00a2880 0x7f32f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:35.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.676+0000 7f33077fe640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f32f00a2880 0x7f32f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:35.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.676+0000 7f3304ff9640 1 -- 172.21.15.77:0/3466342495 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f33000cee50 con 0x7f3308109930 2024-03-16T09:44:35.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.676+0000 7f33077fe640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f32f00a2880 0x7f32f00a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32f8005c20 tx=0x7f32f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:35.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:35.681+0000 7f3304ff9640 1 -- 172.21.15.77:0/3466342495 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3300014030 con 0x7f3308109930 2024-03-16T09:44:36.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.018+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f330802d1b0 con 0x7f3308109930 2024-03-16T09:44:36.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.019+0000 7f3304ff9640 1 -- 172.21.15.77:0/3466342495 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f330009bf00 con 0x7f3308109930 2024-03-16T09:44:36.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:44:36.023 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:44:36.023 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:44:36.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.022+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f32f00a2880 msgr2=0x7f32f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.022+0000 7f330db5b640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f32f00a2880 0x7f32f00a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32f8005c20 tx=0x7f32f8000bc0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.022+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308109930 msgr2=0x7f33081a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.022+0000 7f330db5b640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308109930 0x7f33081a5660 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f330000cce0 tx=0x7f330000a620 comp rx=0 tx=0).stop 2024-03-16T09:44:36.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 shutdown_connections 2024-03-16T09:44:36.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f32f00a2880 0x7f32f00a4d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3308109930 0x7f33081a5660 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3308108f70 0x7f330819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 --2- 172.21.15.77:0/3466342495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3308107e20 0x7f330819e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 >> 172.21.15.77:0/3466342495 conn(0x7f3308076df0 msgr2=0x7f330810c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 shutdown_connections 2024-03-16T09:44:36.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.023+0000 7f330db5b640 1 -- 172.21.15.77:0/3466342495 wait complete. 2024-03-16T09:44:36.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T09:44:36.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 -- 172.21.15.77:0/3089384326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701006e0 msgr2=0x7fb570100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 --2- 172.21.15.77:0/3089384326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701006e0 0x7fb570100b00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb560009900 tx=0x7fb56002bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 -- 172.21.15.77:0/3089384326 shutdown_connections 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 --2- 172.21.15.77:0/3089384326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb570102290 0x7fb570108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 --2- 172.21.15.77:0/3089384326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5701018d0 0x7fb570101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 --2- 172.21.15.77:0/3089384326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701006e0 0x7fb570100b00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 -- 172.21.15.77:0/3089384326 >> 172.21.15.77:0/3089384326 conn(0x7fb5700fbc70 msgr2=0x7fb5700fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 -- 172.21.15.77:0/3089384326 shutdown_connections 2024-03-16T09:44:36.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.195+0000 7fb575fc6640 1 -- 172.21.15.77:0/3089384326 wait complete. 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.196+0000 7fb575fc6640 1 Processor -- start 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.196+0000 7fb575fc6640 1 -- start start 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb575fc6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5701006e0 0x7fb570199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb575fc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701018d0 0x7fb57019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb575fc6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb570102290 0x7fb5701a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb575fc6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb57010c230 con 0x7fb5701018d0 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5701006e0 0x7fb570199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb575fc6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb57010c0b0 con 0x7fb5701006e0 2024-03-16T09:44:36.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb575fc6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb57010c3b0 con 0x7fb570102290 2024-03-16T09:44:36.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5701006e0 0x7fb570199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:34518/0 (socket says 172.21.15.77:34518) 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb56effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701018d0 0x7fb57019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb56f7fe640 1 -- 172.21.15.77:0/1594030543 learned_addr learned my addr 172.21.15.77:0/1594030543 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.197+0000 7fb56ffff640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb570102290 0x7fb5701a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.198+0000 7fb56f7fe640 1 -- 172.21.15.77:0/1594030543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb570102290 msgr2=0x7fb5701a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.198+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb570102290 0x7fb5701a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.198+0000 7fb56f7fe640 1 -- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701018d0 msgr2=0x7fb57019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.198+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701018d0 0x7fb57019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.198+0000 7fb56f7fe640 1 -- 172.21.15.77:0/1594030543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5701a18d0 con 0x7fb5701006e0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.198+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5701006e0 0x7fb570199c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb560000c00 tx=0x7fb5600040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.198+0000 7fb56cff9640 1 -- 172.21.15.77:0/1594030543 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb560043020 con 0x7fb5701006e0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.199+0000 7fb56cff9640 1 -- 172.21.15.77:0/1594030543 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5600044c0 con 0x7fb5701006e0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.199+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5701a1b60 con 0x7fb5701006e0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.199+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5701a2100 con 0x7fb5701006e0 2024-03-16T09:44:36.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.199+0000 7fb56cff9640 1 -- 172.21.15.77:0/1594030543 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb560003c00 con 0x7fb5701006e0 2024-03-16T09:44:36.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.203+0000 7fb56cff9640 1 -- 172.21.15.77:0/1594030543 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb560046020 con 0x7fb5701006e0 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.203+0000 7fb578bb2740 1 Processor -- start 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.203+0000 7fb578bb2740 1 -- start start 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.203+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 0x5650e3a08c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a0dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a10840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb578bb2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5650e396d490 con 0x5650e3a09240 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb578bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5650e3915c90 con 0x5650e3a0e3b0 2024-03-16T09:44:36.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb578bb2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5650e3969530 con 0x5650e3a08850 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a0dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a0dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41634/0 (socket says 172.21.15.77:41634) 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 -- 172.21.15.77:0/944119750 learned_addr learned my addr 172.21.15.77:0/944119750 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56cff9640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb5580a2880 0x7fb5580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56cff9640 1 -- 172.21.15.77:0/1594030543 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb5600f0ab0 con 0x7fb5701006e0 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56ffff640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb5580a2880 0x7fb5580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 -- 172.21.15.77:0/944119750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 msgr2=0x5650e3a08c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 0x5650e3a08c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 -- 172.21.15.77:0/944119750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 msgr2=0x5650e3a10840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56effd640 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a10840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a10840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56f7fe640 1 -- 172.21.15.77:0/944119750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650e39d0500 con 0x5650e3a09240 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.204+0000 7fb56effd640 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a10840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:44:36.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.205+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a0dd40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb5600ee500 tx=0x7fb560009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4329332fff1ba75d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.205+0000 7fb56ffff640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb5580a2880 0x7fb5580a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x5650e3a09aa0 tx=0x7fb568009c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.205+0000 7fb555ffb640 1 -- 172.21.15.77:0/944119750 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5600bd230 con 0x5650e3a09240 2024-03-16T09:44:36.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.205+0000 7fb555ffb640 1 -- 172.21.15.77:0/944119750 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5600bdb60 con 0x5650e3a09240 2024-03-16T09:44:36.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.205+0000 7fb555ffb640 1 -- 172.21.15.77:0/944119750 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5600bcdc0 con 0x5650e3a09240 2024-03-16T09:44:36.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 -- 172.21.15.77:0/944119750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 msgr2=0x5650e3a0dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a0dd40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb5600ee500 tx=0x7fb560009260 comp rx=0 tx=0).stop 2024-03-16T09:44:36.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 -- 172.21.15.77:0/944119750 shutdown_connections 2024-03-16T09:44:36.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a10840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a0dd40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 --2- 172.21.15.77:0/944119750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 0x5650e3a08c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 -- 172.21.15.77:0/944119750 >> 172.21.15.77:0/944119750 conn(0x5650e3a03dc0 msgr2=0x5650e3a041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 -- 172.21.15.77:0/944119750 shutdown_connections 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.206+0000 7fb578bb2740 1 -- 172.21.15.77:0/944119750 wait complete. 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 -- start start 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 Processor -- start 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a12df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a0e3b0 0x5650e3a19e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5650e3a11120 con 0x5650e3a09240 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5650e3929230 con 0x5650e3a08850 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5650e3a112a0 con 0x5650e3a0e3b0 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a0e3b0 0x5650e3a19e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a128b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:34534/0 (socket says 172.21.15.77:34534) 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a12df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56f7fe640 1 -- 172.21.15.77:0/2863435382 learned_addr learned my addr 172.21.15.77:0/2863435382 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.207+0000 7fb578bb2740 10 client.0 _ll_register_callbacks cb 0x5650e3965010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56effd640 1 -- 172.21.15.77:0/2863435382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a0e3b0 msgr2=0x5650e3a19e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56effd640 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a0e3b0 0x5650e3a19e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56effd640 1 -- 172.21.15.77:0/2863435382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 msgr2=0x5650e3a128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56effd640 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a128b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56effd640 1 -- 172.21.15.77:0/2863435382 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650e3938100 con 0x5650e3a09240 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb56effd640 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a12df0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb57019b190 tx=0x7fb56400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c0b7f45fddd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.208+0000 7fb54effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb54effd640 1 -- 172.21.15.77:0/2863435382 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56401b470 con 0x5650e3a09240 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb54effd640 1 -- 172.21.15.77:0/2863435382 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb56401b610 con 0x5650e3a09240 2024-03-16T09:44:36.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5600bdb60 con 0x5650e3a09240 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb54effd640 1 -- 172.21.15.77:0/2863435382 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb56401b810 con 0x5650e3a09240 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb578bb2740 10 client.16107 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5650e36813f0 con 0x5650e3a09240 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb54effd640 1 -- 172.21.15.77:0/2863435382 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5600bdb60 con 0x5650e3a09240 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb54effd640 1 -- 172.21.15.77:0/2863435382 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb564012a70 con 0x5650e3a09240 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb578bb2740 10 client.16107 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb578bb2740 10 client.16107 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.209+0000 7fb578bb2740 20 client.16107 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.210+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a1e720 0x5650e3a20ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.210+0000 7fb578bb2740 4 client.16107 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.210+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650e397a9d0 con 0x5650e3a1e720 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.210+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a1e720 0x5650e3a20ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.211+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a1e720 0x5650e3a20ba0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a92efc49f2d25a server_cookie=621bb479095dc2db in_seq=0 out_seq=0 2024-03-16T09:44:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.211+0000 7fb54effd640 10 client.16107 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.211+0000 7fb54effd640 1 -- 172.21.15.77:0/2863435382 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5650e397a9d0 con 0x5650e3a1e720 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.211+0000 7fb54effd640 10 client.16107 handle_command_reply: tid=0 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.211+0000 7fb578bb2740 1 client.16107 shutdown 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.211+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 msgr2=0x5650e3a12df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.211+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a12df0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb57019b190 tx=0x7fb56400bce0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 shutdown_connections 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a0e3b0 0x5650e3a19e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a12df0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a1e720 0x5650e3a20ba0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2863435382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb54effd640 0 client.16107 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 >> 172.21.15.77:0/2863435382 conn(0x5650e3a03dc0 msgr2=0x5650e3a02fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 shutdown_connections 2024-03-16T09:44:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 1 -- 172.21.15.77:0/2863435382 wait complete. 2024-03-16T09:44:36.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.212+0000 7fb578bb2740 20 client.16107 trim_cache size 0 max 16384 2024-03-16T09:44:36.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 Processor -- start 2024-03-16T09:44:36.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 -- start start 2024-03-16T09:44:36.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 0x5650e3a11ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a19df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c010cc0 con 0x5650e3a09240 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c00e590 con 0x5650e3a0e3b0 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb578bb2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c00c3e0 con 0x5650e3a08850 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb56effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a19df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb56effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a19df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41652/0 (socket says 172.21.15.77:41652) 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb56ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb56effd640 1 -- 172.21.15.77:0/2290695499 learned_addr learned my addr 172.21.15.77:0/2290695499 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.234+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 0x5650e3a11ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb56ffff640 1 -- 172.21.15.77:0/2290695499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 msgr2=0x5650e3a11ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb56ffff640 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 0x5650e3a11ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb56ffff640 1 -- 172.21.15.77:0/2290695499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 msgr2=0x5650e3a19df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb56ffff640 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a19df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb56ffff640 1 -- 172.21.15.77:0/2290695499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650e374b7d0 con 0x5650e3a0e3b0 2024-03-16T09:44:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb56ffff640 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a120e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb568004bb0 tx=0x7fb568014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ea69d8ddec9f89c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb5557fa640 1 -- 172.21.15.77:0/2290695499 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb568005940 con 0x5650e3a0e3b0 2024-03-16T09:44:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb5557fa640 1 -- 172.21.15.77:0/2290695499 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb568023070 con 0x5650e3a0e3b0 2024-03-16T09:44:36.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.235+0000 7fb5557fa640 1 -- 172.21.15.77:0/2290695499 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56800f820 con 0x5650e3a0e3b0 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 -- 172.21.15.77:0/2290695499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 msgr2=0x5650e3a120e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a120e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb568004bb0 tx=0x7fb568014000 comp rx=0 tx=0).stop 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 -- 172.21.15.77:0/2290695499 shutdown_connections 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a0e3b0 0x5650e3a120e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a09240 0x5650e3a19df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2290695499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a08850 0x5650e3a11ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 -- 172.21.15.77:0/2290695499 >> 172.21.15.77:0/2290695499 conn(0x5650e3a03dc0 msgr2=0x5650e3a0a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 -- 172.21.15.77:0/2290695499 shutdown_connections 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.236+0000 7fb578bb2740 1 -- 172.21.15.77:0/2290695499 wait complete. 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 -- start start 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 Processor -- start 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a16eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a09240 0x5650e3a173f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a0e3b0 0x5650e3a18400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c00ca50 con 0x5650e3a0e3b0 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c00dec0 con 0x5650e3a08850 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c00d120 con 0x5650e3a09240 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.237+0000 7fb578bb2740 10 client.0 _ll_register_callbacks cb 0x7fb564004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a0e3b0 0x5650e3a18400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a0e3b0 0x5650e3a18400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41660/0 (socket says 172.21.15.77:41660) 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56ffff640 1 -- 172.21.15.77:0/2279332207 learned_addr learned my addr 172.21.15.77:0/2279332207 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56effd640 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a09240 0x5650e3a173f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56effd640 1 -- 172.21.15.77:0/2279332207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 msgr2=0x5650e3a16eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56effd640 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a16eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56effd640 1 -- 172.21.15.77:0/2279332207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a0e3b0 msgr2=0x5650e3a18400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56effd640 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a0e3b0 0x5650e3a18400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56effd640 1 -- 172.21.15.77:0/2279332207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650e3a189f0 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56ffff640 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a0e3b0 0x5650e3a18400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb56effd640 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a09240 0x5650e3a173f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb5640043b0 tx=0x7fb56401b780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb60bdce8b1b4cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.238+0000 7fb54effd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb54effd640 1 -- 172.21.15.77:0/2279332207 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56401b9b0 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb54effd640 1 -- 172.21.15.77:0/2279332207 <== mon.1 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb56401bb50 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb568023070 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb54effd640 1 -- 172.21.15.77:0/2279332207 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb56401bd70 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 10 client.25876 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5650e3a19d60 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb54effd640 1 -- 172.21.15.77:0/2279332207 <== mon.1 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb568023070 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb54effd640 1 -- 172.21.15.77:0/2279332207 <== mon.1 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb5640049b0 con 0x5650e3a09240 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 10 client.25876 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 10 client.25876 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:44:36.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 20 client.25876 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a84ee0 0x5650e3a87300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 4 client.25876 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.239+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5650e3974d20 con 0x5650e3a84ee0 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.240+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a84ee0 0x5650e3a87300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.240+0000 7fb56f7fe640 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a84ee0 0x5650e3a87300 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7c8258b2b145898 server_cookie=252906233c005888 in_seq=0 out_seq=0 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.240+0000 7fb54effd640 10 client.25876 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb54effd640 1 -- 172.21.15.77:0/2279332207 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5650e3974d20 con 0x5650e3a84ee0 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb54effd640 10 client.25876 handle_command_reply: tid=0 2024-03-16T09:44:36.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 client.25876 shutdown 2024-03-16T09:44:36.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a09240 msgr2=0x5650e3a173f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a09240 0x5650e3a173f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb5640043b0 tx=0x7fb56401b780 comp rx=0 tx=0).stop 2024-03-16T09:44:36.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 shutdown_connections 2024-03-16T09:44:36.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650e3a0e3b0 0x5650e3a18400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650e3a09240 0x5650e3a173f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x5650e3a84ee0 0x5650e3a87300 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 --2- 172.21.15.77:0/2279332207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5650e3a08850 0x5650e3a16eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb54effd640 0 client.25876 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 >> 172.21.15.77:0/2279332207 conn(0x5650e3a03dc0 msgr2=0x5650e3a078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 shutdown_connections 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 1 -- 172.21.15.77:0/2279332207 wait complete. 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.241+0000 7fb578bb2740 20 client.25876 trim_cache size 0 max 16384 2024-03-16T09:44:36.249 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:44:36.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb5580a2880 msgr2=0x7fb5580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb5580a2880 0x7fb5580a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x5650e3a09aa0 tx=0x7fb568009c80 comp rx=0 tx=0).stop 2024-03-16T09:44:36.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5701006e0 msgr2=0x7fb570199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5701006e0 0x7fb570199c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb560000c00 tx=0x7fb5600040c0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 shutdown_connections 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb5580a2880 0x7fb5580a4d40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb570102290 0x7fb5701a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5701018d0 0x7fb57019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 --2- 172.21.15.77:0/1594030543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5701006e0 0x7fb570199c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 >> 172.21.15.77:0/1594030543 conn(0x7fb5700fbc70 msgr2=0x7fb5700689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 shutdown_connections 2024-03-16T09:44:36.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.242+0000 7fb575fc6640 1 -- 172.21.15.77:0/1594030543 wait complete. 2024-03-16T09:44:36.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T09:44:36.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 -- 172.21.15.77:0/726534611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd63c1009a0 msgr2=0x7fd63c100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 --2- 172.21.15.77:0/726534611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd63c1009a0 0x7fd63c100e20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd630009860 tx=0x7fd63002bb70 comp rx=0 tx=0).stop 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 -- 172.21.15.77:0/726534611 shutdown_connections 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 --2- 172.21.15.77:0/726534611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd63c101360 0x7fd63c108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 --2- 172.21.15.77:0/726534611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd63c1009a0 0x7fd63c100e20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 --2- 172.21.15.77:0/726534611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c101c70 0x7fd63c102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 -- 172.21.15.77:0/726534611 >> 172.21.15.77:0/726534611 conn(0x7fd63c0fb3d0 msgr2=0x7fd63c0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 -- 172.21.15.77:0/726534611 shutdown_connections 2024-03-16T09:44:36.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.456+0000 7fd640acc640 1 -- 172.21.15.77:0/726534611 wait complete. 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.457+0000 7fd640acc640 1 Processor -- start 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.457+0000 7fd640acc640 1 -- start start 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.457+0000 7fd640acc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c1009a0 0x7fd63c199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd640acc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd63c101360 0x7fd63c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd640acc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd63c101c70 0x7fd63c1a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd640acc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c10c220 con 0x7fd63c1009a0 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd640acc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c10c0a0 con 0x7fd63c101c70 2024-03-16T09:44:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd640acc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c10c3a0 con 0x7fd63c101360 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c1009a0 0x7fd63c199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c1009a0 0x7fd63c199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41680/0 (socket says 172.21.15.77:41680) 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 -- 172.21.15.77:0/2688700256 learned_addr learned my addr 172.21.15.77:0/2688700256 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd639d9b640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd63c101360 0x7fd63c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 -- 172.21.15.77:0/2688700256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd63c101360 msgr2=0x7fd63c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd63c101360 0x7fd63c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 -- 172.21.15.77:0/2688700256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd63c101c70 msgr2=0x7fd63c1a1270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd63c101c70 0x7fd63c1a1270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 -- 172.21.15.77:0/2688700256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c1a1910 con 0x7fd63c1009a0 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.458+0000 7fd63a59c640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c1009a0 0x7fd63c199d00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00e9a0 tx=0x7fd62c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.459+0000 7fd62b7fe640 1 -- 172.21.15.77:0/2688700256 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c00cd80 con 0x7fd63c1009a0 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.459+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63c1a1c00 con 0x7fd63c1009a0 2024-03-16T09:44:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.459+0000 7fd62b7fe640 1 -- 172.21.15.77:0/2688700256 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c0108e0 con 0x7fd63c1009a0 2024-03-16T09:44:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.459+0000 7fd62b7fe640 1 -- 172.21.15.77:0/2688700256 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c018950 con 0x7fd63c1009a0 2024-03-16T09:44:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.459+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63c1a21a0 con 0x7fd63c1009a0 2024-03-16T09:44:36.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.461+0000 7fd62b7fe640 1 -- 172.21.15.77:0/2688700256 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd62c018af0 con 0x7fd63c1009a0 2024-03-16T09:44:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.462+0000 7fd64378b740 1 Processor -- start 2024-03-16T09:44:36.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.462+0000 7fd62b7fe640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd6240a2800 0x7fd6240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.462+0000 7fd64378b740 1 -- start start 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.462+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd6240a2800 0x7fd6240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd62b7fe640 1 -- 172.21.15.77:0/2688700256 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd62c0ceeb0 con 0x7fd63c1009a0 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd64378b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd64378b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd64378b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3ba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd64378b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5642df317490 con 0x5642df3b83b0 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd64378b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5642df2bfc90 con 0x5642df3b2850 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd64378b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5642df313530 con 0x5642df3b3240 2024-03-16T09:44:36.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd63ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3ba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd6240a2800 0x7fd6240a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd634005fd0 tx=0x7fd634009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd63ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3ba840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41688/0 (socket says 172.21.15.77:41688) 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd63ad9d640 1 -- 172.21.15.77:0/844749501 learned_addr learned my addr 172.21.15.77:0/844749501 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.463+0000 7fd639d9b640 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.464+0000 7fd639d9b640 1 -- 172.21.15.77:0/844749501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 msgr2=0x5642df3b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.464+0000 7fd639d9b640 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.464+0000 7fd639d9b640 1 -- 172.21.15.77:0/844749501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 msgr2=0x5642df3ba840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.464+0000 7fd639d9b640 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3ba840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.464+0000 7fd639d9b640 1 -- 172.21.15.77:0/844749501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642df37a500 con 0x5642df3b3240 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.464+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3ba840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.464+0000 7fd639d9b640 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3b7d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd630037040 tx=0x7fd630009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c28240586096f971 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.465+0000 7fd628ff9640 1 -- 172.21.15.77:0/844749501 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd630043020 con 0x5642df3b3240 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.465+0000 7fd628ff9640 1 -- 172.21.15.77:0/844749501 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6300346c0 con 0x5642df3b3240 2024-03-16T09:44:36.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.465+0000 7fd628ff9640 1 -- 172.21.15.77:0/844749501 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd630039040 con 0x5642df3b3240 2024-03-16T09:44:36.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 -- 172.21.15.77:0/844749501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 msgr2=0x5642df3b7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3b7d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd630037040 tx=0x7fd630009490 comp rx=0 tx=0).stop 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 -- 172.21.15.77:0/844749501 shutdown_connections 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3ba840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3b7d40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 --2- 172.21.15.77:0/844749501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 -- 172.21.15.77:0/844749501 >> 172.21.15.77:0/844749501 conn(0x5642df3addc0 msgr2=0x5642df3ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 -- 172.21.15.77:0/844749501 shutdown_connections 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 -- 172.21.15.77:0/844749501 wait complete. 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.466+0000 7fd64378b740 1 -- start start 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 1 Processor -- start 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3bce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3bd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5642df3bb300 con 0x5642df3b3240 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5642df2d3230 con 0x5642df3b2850 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5642df3bb480 con 0x5642df3b83b0 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd64378b740 10 client.0 _ll_register_callbacks cb 0x5642df30f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd639d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3bd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd639d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3bd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41690/0 (socket says 172.21.15.77:41690) 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd639d9b640 1 -- 172.21.15.77:0/2312304405 learned_addr learned my addr 172.21.15.77:0/2312304405 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.467+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd639d9b640 1 -- 172.21.15.77:0/2312304405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 msgr2=0x5642df3c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd639d9b640 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd639d9b640 1 -- 172.21.15.77:0/2312304405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 msgr2=0x5642df3bce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd639d9b640 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3bce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd639d9b640 1 -- 172.21.15.77:0/2312304405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642df2e2100 con 0x5642df3b3240 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd639d9b640 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3bd3a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd630041000 tx=0x7fd630034df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79566349c7c95619 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd621ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd621ffb640 1 -- 172.21.15.77:0/2312304405 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd630043030 con 0x5642df3b3240 2024-03-16T09:44:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd621ffb640 1 -- 172.21.15.77:0/2312304405 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd630004450 con 0x5642df3b3240 2024-03-16T09:44:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.468+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6300346c0 con 0x5642df3b3240 2024-03-16T09:44:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.469+0000 7fd621ffb640 1 -- 172.21.15.77:0/2312304405 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd63005a050 con 0x5642df3b3240 2024-03-16T09:44:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.469+0000 7fd64378b740 10 client.16125 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:36.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.469+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5642df02b3f0 con 0x5642df3b3240 2024-03-16T09:44:36.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd621ffb640 1 -- 172.21.15.77:0/2312304405 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6300346c0 con 0x5642df3b3240 2024-03-16T09:44:36.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd621ffb640 1 -- 172.21.15.77:0/2312304405 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd630003af0 con 0x5642df3b3240 2024-03-16T09:44:36.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd64378b740 10 client.16125 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd64378b740 10 client.16125 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd64378b740 20 client.16125 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd64378b740 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df3c8860 0x5642df3cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd64378b740 4 client.16125 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.470+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642df3249d0 con 0x5642df3c8860 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.471+0000 7fd63a59c640 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df3c8860 0x5642df3cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.472+0000 7fd63a59c640 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df3c8860 0x5642df3cac50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=591139740f62597d server_cookie=983d68c36e5ebab7 in_seq=0 out_seq=0 2024-03-16T09:44:36.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.472+0000 7fd621ffb640 10 client.16125 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.473+0000 7fd621ffb640 1 -- 172.21.15.77:0/2312304405 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642df3249d0 con 0x5642df3c8860 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.473+0000 7fd621ffb640 10 client.16125 handle_command_reply: tid=0 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.473+0000 7fd64378b740 1 client.16125 shutdown 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 msgr2=0x5642df3bd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3bd3a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd630041000 tx=0x7fd630034df0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 shutdown_connections 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df3c8860 0x5642df3cac50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd621ffb640 0 client.16125 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:36.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3bd3a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 --2- 172.21.15.77:0/2312304405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3bce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 >> 172.21.15.77:0/2312304405 conn(0x5642df3addc0 msgr2=0x5642df3acfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 shutdown_connections 2024-03-16T09:44:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 1 -- 172.21.15.77:0/2312304405 wait complete. 2024-03-16T09:44:36.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.474+0000 7fd64378b740 20 client.16125 trim_cache size 0 max 16384 2024-03-16T09:44:36.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.492+0000 7fd64378b740 1 Processor -- start 2024-03-16T09:44:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.492+0000 7fd64378b740 1 -- start start 2024-03-16T09:44:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd64378b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd64378b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd64378b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd64378b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1ad6e0 con 0x5642df3b3240 2024-03-16T09:44:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd64378b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1aafb0 con 0x5642df3b2850 2024-03-16T09:44:36.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd64378b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1a8e00 con 0x5642df3b83b0 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd639d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd63a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd639d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41702/0 (socket says 172.21.15.77:41702) 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd639d9b640 1 -- 172.21.15.77:0/900628171 learned_addr learned my addr 172.21.15.77:0/900628171 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd63a59c640 1 -- 172.21.15.77:0/900628171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 msgr2=0x5642df3c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.493+0000 7fd63a59c640 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.494+0000 7fd63a59c640 1 -- 172.21.15.77:0/900628171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 msgr2=0x5642df3b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.494+0000 7fd63a59c640 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.494+0000 7fd63a59c640 1 -- 172.21.15.77:0/900628171 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642df0f57d0 con 0x5642df3b2850 2024-03-16T09:44:36.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.494+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:44:36.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.495+0000 7fd63a59c640 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b48f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5642df3b3aa0 tx=0x7fd62c099950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81211b70449e8e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.495+0000 7fd623fff640 1 -- 172.21.15.77:0/900628171 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c009490 con 0x5642df3b2850 2024-03-16T09:44:36.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.495+0000 7fd623fff640 1 -- 172.21.15.77:0/900628171 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c0870c0 con 0x5642df3b2850 2024-03-16T09:44:36.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.495+0000 7fd623fff640 1 -- 172.21.15.77:0/900628171 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c09cb90 con 0x5642df3b2850 2024-03-16T09:44:36.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.496+0000 7fd64378b740 1 -- 172.21.15.77:0/900628171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 msgr2=0x5642df3b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.497+0000 7fd64378b740 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b48f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5642df3b3aa0 tx=0x7fd62c099950 comp rx=0 tx=0).stop 2024-03-16T09:44:36.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.498+0000 7fd64378b740 1 -- 172.21.15.77:0/900628171 shutdown_connections 2024-03-16T09:44:36.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.498+0000 7fd64378b740 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b83b0 0x5642df3c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.498+0000 7fd64378b740 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b3240 0x5642df3b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.498+0000 7fd64378b740 1 --2- 172.21.15.77:0/900628171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3b48f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.498+0000 7fd64378b740 1 -- 172.21.15.77:0/900628171 >> 172.21.15.77:0/900628171 conn(0x5642df3addc0 msgr2=0x5642df3b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.498+0000 7fd64378b740 1 -- 172.21.15.77:0/900628171 shutdown_connections 2024-03-16T09:44:36.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.498+0000 7fd64378b740 1 -- 172.21.15.77:0/900628171 wait complete. 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 -- start start 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 Processor -- start 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3c1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1a9470 con 0x5642df3b83b0 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1aa8e0 con 0x5642df3b2850 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.499+0000 7fd64378b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1a9b40 con 0x5642df3b3240 2024-03-16T09:44:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd64378b740 10 client.0 _ll_register_callbacks cb 0x7fd634014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:36.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd639d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3c1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd639d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3c1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:47946/0 (socket says 172.21.15.77:47946) 2024-03-16T09:44:36.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:34608/0 (socket says 172.21.15.77:34608) 2024-03-16T09:44:36.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd639d9b640 1 -- 172.21.15.77:0/3753595196 learned_addr learned my addr 172.21.15.77:0/3753595196 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63ad9d640 1 -- 172.21.15.77:0/3753595196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 msgr2=0x5642df3c1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3c1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63ad9d640 1 -- 172.21.15.77:0/3753595196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 msgr2=0x5642df3c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.500+0000 7fd63ad9d640 1 -- 172.21.15.77:0/3753595196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642df3c3110 con 0x5642df3b83b0 2024-03-16T09:44:36.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd63ad9d640 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3c2b20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd634010bf0 tx=0x7fd634004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1973eb8faebcba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd621ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd621ffb640 1 -- 172.21.15.77:0/3753595196 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63401ce30 con 0x5642df3b83b0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd621ffb640 1 -- 172.21.15.77:0/3753595196 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd6340263f0 con 0x5642df3b83b0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd62c0870c0 con 0x5642df3b83b0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd621ffb640 1 -- 172.21.15.77:0/3753595196 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd6340265f0 con 0x5642df3b83b0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd64378b740 10 client.16131 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5642df3c4400 con 0x5642df3b83b0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.502+0000 7fd621ffb640 1 -- 172.21.15.77:0/3753595196 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c0870c0 con 0x5642df3b83b0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd621ffb640 1 -- 172.21.15.77:0/3753595196 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd634032800 con 0x5642df3b83b0 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd64378b740 10 client.16131 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd64378b740 10 client.16131 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd64378b740 20 client.16131 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd64378b740 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df42f0a0 0x5642df4314c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd64378b740 4 client.16131 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:44:36.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642df31ed20 con 0x5642df42f0a0 2024-03-16T09:44:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.503+0000 7fd63a59c640 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df42f0a0 0x5642df4314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.505+0000 7fd63a59c640 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df42f0a0 0x5642df4314c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f638f5795217f052 server_cookie=47bd08a2f811384d in_seq=0 out_seq=0 2024-03-16T09:44:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.505+0000 7fd621ffb640 10 client.16131 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.505+0000 7fd621ffb640 1 -- 172.21.15.77:0/3753595196 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642df31ed20 con 0x5642df42f0a0 2024-03-16T09:44:36.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.505+0000 7fd621ffb640 10 client.16131 handle_command_reply: tid=0 2024-03-16T09:44:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.505+0000 7fd64378b740 1 client.16131 shutdown 2024-03-16T09:44:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.505+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 msgr2=0x5642df3c2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.505+0000 7fd64378b740 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3c2b20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd634010bf0 tx=0x7fd634004ab0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 shutdown_connections 2024-03-16T09:44:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x5642df42f0a0 0x5642df4314c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5642df3b83b0 0x5642df3c2b20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5642df3b3240 0x5642df3c1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 --2- 172.21.15.77:0/3753595196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5642df3b2850 0x5642df3c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 >> 172.21.15.77:0/3753595196 conn(0x5642df3addc0 msgr2=0x5642df3b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd621ffb640 0 client.16131 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:44:36.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 shutdown_connections 2024-03-16T09:44:36.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 1 -- 172.21.15.77:0/3753595196 wait complete. 2024-03-16T09:44:36.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.506+0000 7fd64378b740 20 client.16131 trim_cache size 0 max 16384 2024-03-16T09:44:36.511 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:44:36.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd6240a2800 msgr2=0x7fd6240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd6240a2800 0x7fd6240a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd634005fd0 tx=0x7fd634009500 comp rx=0 tx=0).stop 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c1009a0 msgr2=0x7fd63c199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c1009a0 0x7fd63c199d00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00e9a0 tx=0x7fd62c00ee60 comp rx=0 tx=0).stop 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 shutdown_connections 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fd6240a2800 0x7fd6240a4cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd63c101c70 0x7fd63c1a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd63c101360 0x7fd63c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 --2- 172.21.15.77:0/2688700256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c1009a0 0x7fd63c199d00 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 >> 172.21.15.77:0/2688700256 conn(0x7fd63c0fb3d0 msgr2=0x7fd63c0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 shutdown_connections 2024-03-16T09:44:36.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.507+0000 7fd640acc640 1 -- 172.21.15.77:0/2688700256 wait complete. 2024-03-16T09:44:36.529 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.703+0000 7f2e1913b640 1 -- 172.21.15.77:0/2029738480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e14107500 msgr2=0x7f2e14109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.703+0000 7f2e1913b640 1 --2- 172.21.15.77:0/2029738480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e14107500 0x7f2e14109910 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009ca0 tx=0x7f2e0c02c120 comp rx=0 tx=0).stop 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.703+0000 7f2e1913b640 1 -- 172.21.15.77:0/2029738480 shutdown_connections 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.704+0000 7f2e1913b640 1 --2- 172.21.15.77:0/2029738480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e14107500 0x7f2e14109910 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.704+0000 7f2e1913b640 1 --2- 172.21.15.77:0/2029738480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e14104bd0 0x7f2e14106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.704+0000 7f2e1913b640 1 --2- 172.21.15.77:0/2029738480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e141022a0 0x7f2e14104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.704+0000 7f2e1913b640 1 -- 172.21.15.77:0/2029738480 >> 172.21.15.77:0/2029738480 conn(0x7f2e140fbc70 msgr2=0x7f2e140fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.704+0000 7f2e1913b640 1 -- 172.21.15.77:0/2029738480 shutdown_connections 2024-03-16T09:44:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.704+0000 7f2e1913b640 1 -- 172.21.15.77:0/2029738480 wait complete. 2024-03-16T09:44:36.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.705+0000 7f2e1913b640 1 Processor -- start 2024-03-16T09:44:36.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.705+0000 7f2e1913b640 1 -- start start 2024-03-16T09:44:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e1913b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e141022a0 0x7f2e14197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e1913b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e14104bd0 0x7f2e14198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e1913b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e14107500 0x7f2e1419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e1913b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1410d420 con 0x7f2e141022a0 2024-03-16T09:44:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e1913b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1410d2a0 con 0x7f2e14104bd0 2024-03-16T09:44:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e1913b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1410d5a0 con 0x7f2e14107500 2024-03-16T09:44:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e12d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e141022a0 0x7f2e14197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e12d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e141022a0 0x7f2e14197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41714/0 (socket says 172.21.15.77:41714) 2024-03-16T09:44:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.706+0000 7f2e12d9d640 1 -- 172.21.15.77:0/1432944823 learned_addr learned my addr 172.21.15.77:0/1432944823 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.707+0000 7f2e12d9d640 1 -- 172.21.15.77:0/1432944823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e14107500 msgr2=0x7f2e1419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.707+0000 7f2e12d9d640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e14107500 0x7f2e1419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.707+0000 7f2e12d9d640 1 -- 172.21.15.77:0/1432944823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e14104bd0 msgr2=0x7f2e14198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.707+0000 7f2e12d9d640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e14104bd0 0x7f2e14198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.707+0000 7f2e12d9d640 1 -- 172.21.15.77:0/1432944823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e14100190 con 0x7f2e141022a0 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.707+0000 7f2e12d9d640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e141022a0 0x7f2e14197b30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400b450 tx=0x7f2e0400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.707+0000 7f2dfbfff640 1 -- 172.21.15.77:0/1432944823 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04015020 con 0x7f2e141022a0 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.708+0000 7f2dfbfff640 1 -- 172.21.15.77:0/1432944823 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e04011d40 con 0x7f2e141022a0 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.708+0000 7f2dfbfff640 1 -- 172.21.15.77:0/1432944823 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0400f340 con 0x7f2e141022a0 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.708+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e14100480 con 0x7f2e141022a0 2024-03-16T09:44:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.708+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e14100a00 con 0x7f2e141022a0 2024-03-16T09:44:36.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.712+0000 7f2e1be08740 1 Processor -- start 2024-03-16T09:44:36.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.714+0000 7f2e1be08740 1 -- start start 2024-03-16T09:44:36.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf174480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf179550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf179bc0 0x55ffdf17c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1be08740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ffdf0d0490 con 0x55ffdf174a50 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1be08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ffdf078c90 con 0x55ffdf174060 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1be08740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ffdf0cc530 con 0x55ffdf179bc0 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf174480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf174480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:34626/0 (socket says 172.21.15.77:34626) 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 -- 172.21.15.77:0/1407271913 learned_addr learned my addr 172.21.15.77:0/1407271913 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2dfbfff640 1 -- 172.21.15.77:0/1432944823 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2e04018020 con 0x7f2e141022a0 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 -- 172.21.15.77:0/1407271913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf179bc0 msgr2=0x55ffdf17c050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 --2- 172.21.15.77:0/1407271913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf179bc0 0x55ffdf17c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 -- 172.21.15.77:0/1407271913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 msgr2=0x55ffdf179550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 --2- 172.21.15.77:0/1407271913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf179550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.715+0000 7f2e1359e640 1 -- 172.21.15.77:0/1407271913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffdf133500 con 0x55ffdf174060 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.716+0000 7f2e1359e640 1 --2- 172.21.15.77:0/1407271913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf174480 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009bf0 tx=0x7f2e0c02cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbbf0648c5729dff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.716+0000 7f2df97fa640 1 -- 172.21.15.77:0/1407271913 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c041020 con 0x55ffdf174060 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.716+0000 7f2dfbfff640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2dfc0a2880 0x7f2dfc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.716+0000 7f2df97fa640 1 -- 172.21.15.77:0/1407271913 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e0c003e20 con 0x55ffdf174060 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.716+0000 7f2df97fa640 1 -- 172.21.15.77:0/1407271913 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c039040 con 0x55ffdf174060 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.716+0000 7f2e12d9d640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2dfc0a2880 0x7f2dfc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.716+0000 7f2dfbfff640 1 -- 172.21.15.77:0/1432944823 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2e040ce570 con 0x7f2e141022a0 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 -- 172.21.15.77:0/1407271913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 msgr2=0x55ffdf174480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e12d9d640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2dfc0a2880 0x7f2dfc0a4d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009fd0 tx=0x7f2e04066070 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1407271913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf174480 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009bf0 tx=0x7f2e0c02cdc0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 -- 172.21.15.77:0/1407271913 shutdown_connections 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1407271913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf179bc0 0x55ffdf17c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1407271913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf179550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1407271913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf174480 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 -- 172.21.15.77:0/1407271913 >> 172.21.15.77:0/1407271913 conn(0x55ffdf16f5d0 msgr2=0x55ffdf16f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 -- 172.21.15.77:0/1407271913 shutdown_connections 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.717+0000 7f2e1be08740 1 -- 172.21.15.77:0/1407271913 wait complete. 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 -- start start 2024-03-16T09:44:36.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 Processor -- start 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf17e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf17ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf179bc0 0x55ffdf185a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ffdf17cb60 con 0x55ffdf179bc0 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ffdf08c230 con 0x55ffdf174060 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ffdf17cce0 con 0x55ffdf174a50 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.718+0000 7f2e1be08740 10 client.0 _ll_register_callbacks cb 0x55ffdf0c8010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf179bc0 0x55ffdf185a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf179bc0 0x55ffdf185a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41728/0 (socket says 172.21.15.77:41728) 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 -- 172.21.15.77:0/1561596698 learned_addr learned my addr 172.21.15.77:0/1561596698 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf17e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1259c640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf17ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 -- 172.21.15.77:0/1561596698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 msgr2=0x55ffdf17ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf17ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 -- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 msgr2=0x55ffdf17e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf17e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 -- 172.21.15.77:0/1561596698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffdf09b100 con 0x55ffdf179bc0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1259c640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf17ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1359e640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf179bc0 0x55ffdf185a50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c02cfd0 tx=0x7f2e0c004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368ce6dcdede8e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2df6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2df6ff5640 1 -- 172.21.15.77:0/1561596698 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c050020 con 0x55ffdf179bc0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2df6ff5640 1 -- 172.21.15.77:0/1561596698 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2e0c032890 con 0x55ffdf179bc0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.719+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e0c003e20 con 0x55ffdf179bc0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2df6ff5640 1 -- 172.21.15.77:0/1561596698 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2e0c041070 con 0x55ffdf179bc0 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 10 client.16140 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ffdede43f0 con 0x55ffdf179bc0 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2df6ff5640 1 -- 172.21.15.77:0/1561596698 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e0c003e20 con 0x55ffdf179bc0 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2df6ff5640 1 -- 172.21.15.77:0/1561596698 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2e0c03c030 con 0x55ffdf179bc0 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 10 client.16140 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 20 client.16140 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf189f40 0x55ffdf18c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 4 client.16140 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffdf0dd9d0 con 0x55ffdf189f40 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.720+0000 7f2e1259c640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf189f40 0x55ffdf18c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.721+0000 7f2e1259c640 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf189f40 0x55ffdf18c330 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff906e36c829d92e server_cookie=ecfb31070ab6863e in_seq=0 out_seq=0 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.721+0000 7f2df6ff5640 10 client.16140 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2df6ff5640 1 -- 172.21.15.77:0/1561596698 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ffdf0dd9d0 con 0x55ffdf189f40 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2df6ff5640 10 client.16140 handle_command_reply: tid=0 2024-03-16T09:44:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 client.16140 shutdown 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf179bc0 msgr2=0x55ffdf185a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf179bc0 0x55ffdf185a50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c02cfd0 tx=0x7f2e0c004840 comp rx=0 tx=0).stop 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 shutdown_connections 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf189f40 0x55ffdf18c330 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf179bc0 0x55ffdf185a50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2df6ff5640 0 client.16140 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf17ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 --2- 172.21.15.77:0/1561596698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf174060 0x55ffdf17e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 >> 172.21.15.77:0/1561596698 conn(0x55ffdf16f5d0 msgr2=0x55ffdf165fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 shutdown_connections 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.722+0000 7f2e1be08740 1 -- 172.21.15.77:0/1561596698 wait complete. 2024-03-16T09:44:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.723+0000 7f2e1be08740 20 client.16140 trim_cache size 0 max 16384 2024-03-16T09:44:36.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 Processor -- start 2024-03-16T09:44:36.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 -- start start 2024-03-16T09:44:36.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174060 0x55ffdf175eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf1763f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf185970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e141a9300 con 0x55ffdf174060 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2e141a6bd0 con 0x55ffdf179bc0 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1be08740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2e141a4a20 con 0x55ffdf174a50 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174060 0x55ffdf175eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174060 0x55ffdf175eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41730/0 (socket says 172.21.15.77:41730) 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 -- 172.21.15.77:0/3382150990 learned_addr learned my addr 172.21.15.77:0/3382150990 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf185970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 -- 172.21.15.77:0/3382150990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 msgr2=0x55ffdf1763f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf1763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 -- 172.21.15.77:0/3382150990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 msgr2=0x55ffdf185970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.744+0000 7f2e1359e640 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf185970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.745+0000 7f2e1359e640 1 -- 172.21.15.77:0/3382150990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffdeeae7d0 con 0x55ffdf174060 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.745+0000 7f2e1359e640 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174060 0x55ffdf175eb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c02c660 tx=0x7f2e0c02cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d327c1f06531f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.745+0000 7f2df8ff9640 1 -- 172.21.15.77:0/3382150990 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c039030 con 0x55ffdf174060 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.745+0000 7f2df8ff9640 1 -- 172.21.15.77:0/3382150990 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e0c003db0 con 0x55ffdf174060 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.745+0000 7f2df8ff9640 1 -- 172.21.15.77:0/3382150990 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c050070 con 0x55ffdf174060 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.746+0000 7f2e1be08740 1 -- 172.21.15.77:0/3382150990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174060 msgr2=0x55ffdf175eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.746+0000 7f2e1be08740 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174060 0x55ffdf175eb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c02c660 tx=0x7f2e0c02cdc0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.746+0000 7f2e1be08740 1 -- 172.21.15.77:0/3382150990 shutdown_connections 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.747+0000 7f2e1be08740 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf185970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.747+0000 7f2e1be08740 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174a50 0x55ffdf1763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.747+0000 7f2e1be08740 1 --2- 172.21.15.77:0/3382150990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174060 0x55ffdf175eb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.747+0000 7f2e1be08740 1 -- 172.21.15.77:0/3382150990 >> 172.21.15.77:0/3382150990 conn(0x55ffdf16f5d0 msgr2=0x55ffdf17e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.747+0000 7f2e1be08740 1 -- 172.21.15.77:0/3382150990 shutdown_connections 2024-03-16T09:44:36.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.747+0000 7f2e1be08740 1 -- 172.21.15.77:0/3382150990 wait complete. 2024-03-16T09:44:36.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.748+0000 7f2e1be08740 1 -- start start 2024-03-16T09:44:36.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.748+0000 7f2e1be08740 1 Processor -- start 2024-03-16T09:44:36.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174060 0x55ffdf1828c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf182e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1be08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf1ee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1be08740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e141a5090 con 0x55ffdf174a50 2024-03-16T09:44:36.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1be08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2e141a6500 con 0x55ffdf179bc0 2024-03-16T09:44:36.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1be08740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2e141a5760 con 0x55ffdf174060 2024-03-16T09:44:36.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1be08740 10 client.0 _ll_register_callbacks cb 0x7f2e0c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf182e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf182e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41736/0 (socket says 172.21.15.77:41736) 2024-03-16T09:44:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.749+0000 7f2e1259c640 1 -- 172.21.15.77:0/2527847606 learned_addr learned my addr 172.21.15.77:0/2527847606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1359e640 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf1ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1259c640 1 -- 172.21.15.77:0/2527847606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174060 msgr2=0x55ffdf1828c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1259c640 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174060 0x55ffdf1828c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1259c640 1 -- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 msgr2=0x55ffdf1ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1259c640 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf1ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1259c640 1 -- 172.21.15.77:0/2527847606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffdf1ee6a0 con 0x55ffdf174a50 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1359e640 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf1ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2e1259c640 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf182e00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800a770 tx=0x7f2e0800ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ca56ed0efab82f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.750+0000 7f2df6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.751+0000 7f2df6ff5640 1 -- 172.21.15.77:0/2527847606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e08013de0 con 0x55ffdf174a50 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.751+0000 7f2df6ff5640 1 -- 172.21.15.77:0/2527847606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2e080160e0 con 0x55ffdf174a50 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.751+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e0c003db0 con 0x55ffdf174a50 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.751+0000 7f2df6ff5640 1 -- 172.21.15.77:0/2527847606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2e080162e0 con 0x55ffdf174a50 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.751+0000 7f2e1be08740 10 client.16146 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:36.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.751+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ffdf1efaa0 con 0x55ffdf174a50 2024-03-16T09:44:36.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.752+0000 7f2df6ff5640 1 -- 172.21.15.77:0/2527847606 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e0c003db0 con 0x55ffdf174a50 2024-03-16T09:44:36.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.752+0000 7f2df6ff5640 1 -- 172.21.15.77:0/2527847606 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2e0801fc20 con 0x55ffdf174a50 2024-03-16T09:44:36.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.752+0000 7f2e1be08740 10 client.16146 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:36.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.752+0000 7f2e1be08740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:36.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.752+0000 7f2e1be08740 20 client.16146 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:36.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.752+0000 7f2e1be08740 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf1f0430 0x55ffdf1f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:36.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.752+0000 7f2e1be08740 4 client.16146 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:44:36.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.753+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffdf0d7d20 con 0x55ffdf1f0430 2024-03-16T09:44:36.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.755+0000 7f2e1359e640 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf1f0430 0x55ffdf1f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:36.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.757+0000 7f2e1359e640 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf1f0430 0x55ffdf1f2850 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=430b7cd7c235a829 server_cookie=d8e0ed7fca9095f4 in_seq=0 out_seq=0 2024-03-16T09:44:36.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.757+0000 7f2df6ff5640 10 client.16146 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:36.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.758+0000 7f2df6ff5640 1 -- 172.21.15.77:0/2527847606 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffdf0d7d20 con 0x55ffdf1f0430 2024-03-16T09:44:36.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.758+0000 7f2df6ff5640 10 client.16146 handle_command_reply: tid=0 2024-03-16T09:44:36.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.758+0000 7f2e1be08740 1 client.16146 shutdown 2024-03-16T09:44:36.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.758+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 msgr2=0x55ffdf182e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.758+0000 7f2e1be08740 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf182e00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800a770 tx=0x7f2e0800ae70 comp rx=0 tx=0).stop 2024-03-16T09:44:36.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 shutdown_connections 2024-03-16T09:44:36.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ffdf1f0430 0x55ffdf1f2850 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ffdf179bc0 0x55ffdf1ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffdf174a50 0x55ffdf182e00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 --2- 172.21.15.77:0/2527847606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffdf174060 0x55ffdf1828c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 >> 172.21.15.77:0/2527847606 conn(0x55ffdf16f5d0 msgr2=0x55ffdf17e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2df6ff5640 0 client.16146 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:36.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 shutdown_connections 2024-03-16T09:44:36.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 1 -- 172.21.15.77:0/2527847606 wait complete. 2024-03-16T09:44:36.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.759+0000 7f2e1be08740 20 client.16146 trim_cache size 0 max 16384 2024-03-16T09:44:36.763 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2dfc0a2880 msgr2=0x7f2dfc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2dfc0a2880 0x7f2dfc0a4d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009fd0 tx=0x7f2e04066070 comp rx=0 tx=0).stop 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e141022a0 msgr2=0x7f2e14197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e141022a0 0x7f2e14197b30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2e0400b450 tx=0x7f2e0400b910 comp rx=0 tx=0).stop 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 shutdown_connections 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f2dfc0a2880 0x7f2dfc0a4d40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e14107500 0x7f2e1419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e14104bd0 0x7f2e14198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 --2- 172.21.15.77:0/1432944823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e141022a0 0x7f2e14197b30 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:36.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 >> 172.21.15.77:0/1432944823 conn(0x7f2e140fbc70 msgr2=0x7f2e14102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:36.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 shutdown_connections 2024-03-16T09:44:36.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:36.760+0000 7f2e1913b640 1 -- 172.21.15.77:0/1432944823 wait complete. 2024-03-16T09:44:36.946 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:36 smithi077 ceph-mon[32171]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 493 MiB/s wr, 131 op/s 2024-03-16T09:44:36.946 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:36 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3466342495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:44:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:36 smithi129 ceph-mon[35841]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 493 MiB/s wr, 131 op/s 2024-03-16T09:44:37.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:36 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3466342495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:44:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:36 smithi119 ceph-mon[35632]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 493 MiB/s wr, 131 op/s 2024-03-16T09:44:37.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:36 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3466342495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:44:37.787 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T09:44:37.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:44:37.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 -- 172.21.15.77:0/1767755293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec1017b0 msgr2=0x7f99ec101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 --2- 172.21.15.77:0/1767755293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec1017b0 0x7f99ec101c30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009900 tx=0x7f99d802bc50 comp rx=0 tx=0).stop 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 -- 172.21.15.77:0/1767755293 shutdown_connections 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 --2- 172.21.15.77:0/1767755293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102170 0x7f99ec069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 --2- 172.21.15.77:0/1767755293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec1017b0 0x7f99ec101c30 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 --2- 172.21.15.77:0/1767755293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99ec1005c0 0x7f99ec1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 -- 172.21.15.77:0/1767755293 >> 172.21.15.77:0/1767755293 conn(0x7f99ec0fbb50 msgr2=0x7f99ec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 -- 172.21.15.77:0/1767755293 shutdown_connections 2024-03-16T09:44:37.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.977+0000 7f99f1af9640 1 -- 172.21.15.77:0/1767755293 wait complete. 2024-03-16T09:44:37.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.978+0000 7f99f1af9640 1 Processor -- start 2024-03-16T09:44:37.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.978+0000 7f99f1af9640 1 -- start start 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.978+0000 7f99f1af9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec1005c0 0x7f99ec199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f1af9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99ec1017b0 0x7f99ec19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f1af9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec102170 0x7f99ec1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f1af9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10f540 con 0x7f99ec102170 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f1af9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10f3c0 con 0x7f99ec1005c0 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f1af9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10f6c0 con 0x7f99ec1017b0 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec102170 0x7f99ec1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec102170 0x7f99ec1a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41762/0 (socket says 172.21.15.77:41762) 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 -- 172.21.15.77:0/2996084439 learned_addr learned my addr 172.21.15.77:0/2996084439 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 -- 172.21.15.77:0/2996084439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99ec1017b0 msgr2=0x7f99ec19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99ec1017b0 0x7f99ec19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 -- 172.21.15.77:0/2996084439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec1005c0 msgr2=0x7f99ec199cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec1005c0 0x7f99ec199cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.979+0000 7f99f12f8640 1 -- 172.21.15.77:0/2996084439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec1a1900 con 0x7f99ec102170 2024-03-16T09:44:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.980+0000 7f99f12f8640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec102170 0x7f99ec1a1260 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f99e400b990 tx=0x7f99e400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.980+0000 7f99e9ffb640 1 -- 172.21.15.77:0/2996084439 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4019070 con 0x7f99ec102170 2024-03-16T09:44:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.980+0000 7f99e9ffb640 1 -- 172.21.15.77:0/2996084439 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e400cba0 con 0x7f99ec102170 2024-03-16T09:44:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.980+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ec1a1bf0 con 0x7f99ec102170 2024-03-16T09:44:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.980+0000 7f99e9ffb640 1 -- 172.21.15.77:0/2996084439 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e40075d0 con 0x7f99ec102170 2024-03-16T09:44:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.980+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ec1a2160 con 0x7f99ec102170 2024-03-16T09:44:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.983+0000 7f99e9ffb640 1 -- 172.21.15.77:0/2996084439 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f99e401c070 con 0x7f99ec102170 2024-03-16T09:44:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 Processor -- start 2024-03-16T09:44:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 -- start start 2024-03-16T09:44:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 0x55ff9ecd7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ecdd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ecdfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ff9ec3c0d0 con 0x55ff9ecd8540 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ff9ebe4c70 con 0x55ff9ecdd6b0 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f5a29740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ff9ec38170 con 0x55ff9ecd7b50 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ecdd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ecdd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41776/0 (socket says 172.21.15.77:41776) 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 -- 172.21.15.77:0/432396839 learned_addr learned my addr 172.21.15.77:0/432396839 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99e9ffb640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f99dc0a2880 0x7f99dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 -- 172.21.15.77:0/432396839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 msgr2=0x55ff9ecd7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 --2- 172.21.15.77:0/432396839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 0x55ff9ecd7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 -- 172.21.15.77:0/432396839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 msgr2=0x55ff9ecdfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 --2- 172.21.15.77:0/432396839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ecdfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.984+0000 7f99f0af7640 1 -- 172.21.15.77:0/432396839 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff9ea187e0 con 0x55ff9ecd8540 2024-03-16T09:44:37.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.985+0000 7f99e9ffb640 1 -- 172.21.15.77:0/2996084439 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f99e40cdf90 con 0x7f99ec102170 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.985+0000 7f99f12f8640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f99dc0a2880 0x7f99dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.985+0000 7f99f0af7640 1 --2- 172.21.15.77:0/432396839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ecdd040 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009740 tx=0x7f99e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107cee12b1f09b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.985+0000 7f99d6ffd640 1 -- 172.21.15.77:0/432396839 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0015020 con 0x55ff9ecd8540 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.985+0000 7f99d6ffd640 1 -- 172.21.15.77:0/432396839 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e0009e50 con 0x55ff9ecd8540 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.985+0000 7f99d6ffd640 1 -- 172.21.15.77:0/432396839 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0010380 con 0x55ff9ecd8540 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 -- 172.21.15.77:0/432396839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 msgr2=0x55ff9ecdd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 --2- 172.21.15.77:0/432396839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ecdd040 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009740 tx=0x7f99e0009c00 comp rx=0 tx=0).stop 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f12f8640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f99dc0a2880 0x7f99dc0a4d40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f99e4004b10 tx=0x7f99e4004a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 -- 172.21.15.77:0/432396839 shutdown_connections 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 --2- 172.21.15.77:0/432396839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ecdfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 --2- 172.21.15.77:0/432396839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ecdd040 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 --2- 172.21.15.77:0/432396839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 0x55ff9ecd7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 -- 172.21.15.77:0/432396839 >> 172.21.15.77:0/432396839 conn(0x55ff9ec9fc50 msgr2=0x55ff9ecd3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:37.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 -- 172.21.15.77:0/432396839 shutdown_connections 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 -- 172.21.15.77:0/432396839 wait complete. 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 -- start start 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.986+0000 7f99f5a29740 1 Processor -- start 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 0x55ff9ece2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f5a29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ff9ece0660 con 0x55ff9ecd8540 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f5a29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ff9ebf80d0 con 0x55ff9ecdd6b0 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f5a29740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ff9ece07e0 con 0x55ff9ecd7b50 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f5a29740 10 client.0 _ll_register_callbacks cb 0x55ff9ebc1760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41780/0 (socket says 172.21.15.77:41780) 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99f0af7640 1 -- 172.21.15.77:0/3348939881 learned_addr learned my addr 172.21.15.77:0/3348939881 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.987+0000 7f99ebfff640 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:37.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.988+0000 7f99f0af7640 1 -- 172.21.15.77:0/3348939881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 msgr2=0x55ff9ece2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.988+0000 7f99f0af7640 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 0x55ff9ece2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.988+0000 7f99f0af7640 1 -- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 msgr2=0x55ff9ece96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.988+0000 7f99f0af7640 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.988+0000 7f99f0af7640 1 -- 172.21.15.77:0/3348939881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff9ea78f70 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.988+0000 7f99f0af7640 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece2660 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f99e0004810 tx=0x7f99e0011790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706462f25ba86d21 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.988+0000 7f99cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.989+0000 7f99cbfff640 1 -- 172.21.15.77:0/3348939881 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0024470 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.989+0000 7f99cbfff640 1 -- 172.21.15.77:0/3348939881 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f99e0024610 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.989+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e0009e50 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.989+0000 7f99cbfff640 1 -- 172.21.15.77:0/3348939881 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f99e0024810 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.989+0000 7f99f5a29740 10 client.16158 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.989+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ff9e910170 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.989+0000 7f99cbfff640 1 -- 172.21.15.77:0/3348939881 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e0009e50 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.990+0000 7f99cbfff640 1 -- 172.21.15.77:0/3348939881 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f99e001aac0 con 0x55ff9ecd8540 2024-03-16T09:44:37.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.990+0000 7f99f5a29740 10 client.16158 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:37.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.990+0000 7f99f5a29740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:37.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.990+0000 7f99f5a29740 20 client.16158 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:37.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.990+0000 7f99f5a29740 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ecefcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:37.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.990+0000 7f99f5a29740 4 client.16158 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:37.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.990+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff9ec49610 con 0x55ff9eced8c0 2024-03-16T09:44:37.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.991+0000 7f99ebfff640 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ecefcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:37.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.994+0000 7f99ebfff640 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ecefcb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=715c1e1168056215 server_cookie=84319a4f184f5afa in_seq=0 out_seq=0 2024-03-16T09:44:37.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.994+0000 7f99cbfff640 10 client.16158 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99cbfff640 1 -- 172.21.15.77:0/3348939881 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff9ec49610 con 0x55ff9eced8c0 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99cbfff640 10 client.16158 handle_command_reply: tid=0 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 client.16158 shutdown 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 msgr2=0x55ff9ece2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece2660 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f99e0004810 tx=0x7f99e0011790 comp rx=0 tx=0).stop 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 shutdown_connections 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ecefcb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99cbfff640 0 client.16158 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece2660 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.996+0000 7f99f5a29740 1 --2- 172.21.15.77:0/3348939881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecd7b50 0x55ff9ece2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.997+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 >> 172.21.15.77:0/3348939881 conn(0x55ff9ec9fc50 msgr2=0x55ff9ecd26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.997+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 shutdown_connections 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.997+0000 7f99f5a29740 1 -- 172.21.15.77:0/3348939881 wait complete. 2024-03-16T09:44:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:37.997+0000 7f99f5a29740 20 client.16158 trim_cache size 0 max 16384 2024-03-16T09:44:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 Processor -- start 2024-03-16T09:44:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 -- start start 2024-03-16T09:44:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 0x55ff9ecd6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ecd6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec1ad640 con 0x55ff9ecd8540 2024-03-16T09:44:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec1aaf10 con 0x55ff9ecd7b50 2024-03-16T09:44:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f5a29740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec1a8d60 con 0x55ff9ecdd6b0 2024-03-16T09:44:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41790/0 (socket says 172.21.15.77:41790) 2024-03-16T09:44:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99f0af7640 1 -- 172.21.15.77:0/1085690381 learned_addr learned my addr 172.21.15.77:0/1085690381 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.028+0000 7f99ebfff640 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ecd6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99f0af7640 1 -- 172.21.15.77:0/1085690381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 msgr2=0x55ff9ecd6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99f0af7640 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ecd6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99f0af7640 1 -- 172.21.15.77:0/1085690381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 msgr2=0x55ff9ecd6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99f0af7640 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 0x55ff9ecd6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99f0af7640 1 -- 172.21.15.77:0/1085690381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff9ecd4210 con 0x55ff9ecd8540 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99f0af7640 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece9660 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f99e001aee0 tx=0x7f99e0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410bebdae72445fa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99d67fc640 1 -- 172.21.15.77:0/1085690381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0033030 con 0x55ff9ecd8540 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99d67fc640 1 -- 172.21.15.77:0/1085690381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e001f070 con 0x55ff9ecd8540 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.029+0000 7f99d67fc640 1 -- 172.21.15.77:0/1085690381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0015040 con 0x55ff9ecd8540 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 -- 172.21.15.77:0/1085690381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 msgr2=0x55ff9ece9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece9660 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f99e001aee0 tx=0x7f99e0009040 comp rx=0 tx=0).stop 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 -- 172.21.15.77:0/1085690381 shutdown_connections 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ecd6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece9660 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1085690381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 0x55ff9ecd6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 -- 172.21.15.77:0/1085690381 >> 172.21.15.77:0/1085690381 conn(0x55ff9ec9fc50 msgr2=0x55ff9ece0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 -- 172.21.15.77:0/1085690381 shutdown_connections 2024-03-16T09:44:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 -- 172.21.15.77:0/1085690381 wait complete. 2024-03-16T09:44:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 -- start start 2024-03-16T09:44:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.031+0000 7f99f5a29740 1 Processor -- start 2024-03-16T09:44:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 0x55ff9ece71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f5a29740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f5a29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec1a93d0 con 0x55ff9ecd8540 2024-03-16T09:44:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f5a29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec1aa840 con 0x55ff9ecd7b50 2024-03-16T09:44:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f0af7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41794/0 (socket says 172.21.15.77:41794) 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f0af7640 1 -- 172.21.15.77:0/1874189339 learned_addr learned my addr 172.21.15.77:0/1874189339 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec1a9aa0 con 0x55ff9ecdd6b0 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99ebfff640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 0x55ff9ece71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.032+0000 7f99f5a29740 10 client.0 _ll_register_callbacks cb 0x7f99e0009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99ebfff640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99f0af7640 1 -- 172.21.15.77:0/1874189339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 msgr2=0x55ff9ece86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99f0af7640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99f0af7640 1 -- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 msgr2=0x55ff9ece71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99f0af7640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 0x55ff9ece71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99f0af7640 1 -- 172.21.15.77:0/1874189339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff9ece8cc0 con 0x55ff9ecd8540 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99ebfff640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99f0af7640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece7730 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f99e00094d0 tx=0x7f99e001b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb932091e235e3fb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.033+0000 7f99cbfff640 1 -- 172.21.15.77:0/1874189339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e003d060 con 0x55ff9ecd8540 2024-03-16T09:44:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.034+0000 7f99cbfff640 1 -- 172.21.15.77:0/1874189339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f99e00249b0 con 0x55ff9ecd8540 2024-03-16T09:44:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.034+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e001f070 con 0x55ff9ecd8540 2024-03-16T09:44:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.034+0000 7f99cbfff640 1 -- 172.21.15.77:0/1874189339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f99e0033070 con 0x55ff9ecd8540 2024-03-16T09:44:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.034+0000 7f99f5a29740 10 client.16164 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.035+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ff9ed5c780 con 0x55ff9ecd8540 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.035+0000 7f99cbfff640 1 -- 172.21.15.77:0/1874189339 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e0006e00 con 0x55ff9ecd8540 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.036+0000 7f99cbfff640 1 -- 172.21.15.77:0/1874189339 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f99e001a430 con 0x55ff9ecd8540 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.036+0000 7f99f5a29740 10 client.16164 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.036+0000 7f99f5a29740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.036+0000 7f99f5a29740 20 client.16164 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.036+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ed5f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.037+0000 7f99f5a29740 4 client.16164 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T09:44:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.037+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f99e001f070 con 0x55ff9eced8c0 2024-03-16T09:44:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.037+0000 7f99ebfff640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ed5f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.038+0000 7f99ebfff640 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ed5f0f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15bf025c876c5c32 server_cookie=c697a314004bbbfa in_seq=0 out_seq=0 2024-03-16T09:44:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.038+0000 7f99cbfff640 10 client.16164 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.040+0000 7f99cbfff640 1 -- 172.21.15.77:0/1874189339 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f99e001f070 con 0x55ff9eced8c0 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.040+0000 7f99cbfff640 10 client.16164 handle_command_reply: tid=0 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.040+0000 7f99f5a29740 1 client.16164 shutdown 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.040+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 msgr2=0x55ff9ece7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.040+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece7730 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f99e00094d0 tx=0x7f99e001b660 comp rx=0 tx=0).stop 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 shutdown_connections 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55ff9eced8c0 0x55ff9ed5f0f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99cbfff640 0 client.16164 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ff9ecdd6b0 0x55ff9ece86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff9ecd8540 0x55ff9ece7730 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 --2- 172.21.15.77:0/1874189339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ff9ecd7b50 0x55ff9ece71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 >> 172.21.15.77:0/1874189339 conn(0x55ff9ec9fc50 msgr2=0x55ff9ecd6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 shutdown_connections 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 1 -- 172.21.15.77:0/1874189339 wait complete. 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.041+0000 7f99f5a29740 20 client.16164 trim_cache size 0 max 16384 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "65f194f6-dcd7-45d1-8d40-1be2a908656a", 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T09:44:38.045 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:44:38.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f99dc0a2880 msgr2=0x7f99dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f99dc0a2880 0x7f99dc0a4d40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f99e4004b10 tx=0x7f99e4004a60 comp rx=0 tx=0).stop 2024-03-16T09:44:38.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec102170 msgr2=0x7f99ec1a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec102170 0x7f99ec1a1260 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f99e400b990 tx=0x7f99e400be50 comp rx=0 tx=0).stop 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 shutdown_connections 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f99dc0a2880 0x7f99dc0a4d40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99ec102170 0x7f99ec1a1260 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99ec1017b0 0x7f99ec19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 --2- 172.21.15.77:0/2996084439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec1005c0 0x7f99ec199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 >> 172.21.15.77:0/2996084439 conn(0x7f99ec0fbb50 msgr2=0x7f99ec1029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 shutdown_connections 2024-03-16T09:44:38.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.042+0000 7f99f1af9640 1 -- 172.21.15.77:0/2996084439 wait complete. 2024-03-16T09:44:38.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:44:38.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:44:38.104 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:38 smithi077 ceph-mon[32171]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 630 MiB/s wr, 165 op/s 2024-03-16T09:44:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 -- 172.21.15.77:0/2032024280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b041034f0 msgr2=0x7f8b04109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/2032024280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b041034f0 0x7f8b04109d80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009cd0 tx=0x7f8afc02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 -- 172.21.15.77:0/2032024280 shutdown_connections 2024-03-16T09:44:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/2032024280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b041034f0 0x7f8b04109d80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/2032024280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b04102b30 0x7f8b04102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/2032024280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b040ffac0 0x7f8b040ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 -- 172.21.15.77:0/2032024280 >> 172.21.15.77:0/2032024280 conn(0x7f8b040fb3d0 msgr2=0x7f8b040fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 -- 172.21.15.77:0/2032024280 shutdown_connections 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.229+0000 7f8b0ab53640 1 -- 172.21.15.77:0/2032024280 wait complete. 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 Processor -- start 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 -- start start 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b040ffac0 0x7f8b04197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b04102b30 0x7f8b04198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b041034f0 0x7f8b0419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0410d410 con 0x7f8b04102b30 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0410d290 con 0x7f8b040ffac0 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b0ab53640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0410d590 con 0x7f8b041034f0 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b04102b30 0x7f8b04198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b04102b30 0x7f8b04198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41814/0 (socket says 172.21.15.77:41814) 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 -- 172.21.15.77:0/1091551258 learned_addr learned my addr 172.21.15.77:0/1091551258 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b088ef640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b040ffac0 0x7f8b04197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 -- 172.21.15.77:0/1091551258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b041034f0 msgr2=0x7f8b0419f0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b041034f0 0x7f8b0419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 -- 172.21.15.77:0/1091551258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b040ffac0 msgr2=0x7f8b04197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b040ffac0 0x7f8b04197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b03fff640 1 -- 172.21.15.77:0/1091551258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0419f730 con 0x7f8b04102b30 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.230+0000 7f8b088ef640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b040ffac0 0x7f8b04197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.231+0000 7f8b03fff640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b04102b30 0x7f8b04198050 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8af000b4a0 tx=0x7f8af000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.231+0000 7f8b01ffb640 1 -- 172.21.15.77:0/1091551258 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af0014070 con 0x7f8b04102b30 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.231+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b0419fa20 con 0x7f8b04102b30 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.231+0000 7f8b01ffb640 1 -- 172.21.15.77:0/1091551258 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af0010ac0 con 0x7f8b04102b30 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.231+0000 7f8b01ffb640 1 -- 172.21.15.77:0/1091551258 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af0014070 con 0x7f8b04102b30 2024-03-16T09:44:38.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.231+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b041a0040 con 0x7f8b04102b30 2024-03-16T09:44:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.235+0000 7f8b0d81e740 1 Processor -- start 2024-03-16T09:44:38.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.235+0000 7f8b0d81e740 1 -- start start 2024-03-16T09:44:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5e2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 0x560dea5e7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5ea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560dea547500 con 0x560dea5e7cd0 2024-03-16T09:44:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560dea4efbe0 con 0x560dea5e2170 2024-03-16T09:44:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560dea5435a0 con 0x560dea5e2b60 2024-03-16T09:44:38.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b01ffb640 1 -- 172.21.15.77:0/1091551258 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8af0017070 con 0x7f8b04102b30 2024-03-16T09:44:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b090f0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5e2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b090f0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5e2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:34748/0 (socket says 172.21.15.77:34748) 2024-03-16T09:44:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.236+0000 7f8b090f0640 1 -- 172.21.15.77:0/4283881672 learned_addr learned my addr 172.21.15.77:0/4283881672 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.237+0000 7f8b088ef640 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 0x560dea5e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.237+0000 7f8b03fff640 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.237+0000 7f8b090f0640 1 -- 172.21.15.77:0/4283881672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 msgr2=0x560dea5e7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.237+0000 7f8b090f0640 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 0x560dea5e7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.237+0000 7f8b090f0640 1 -- 172.21.15.77:0/4283881672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 msgr2=0x560dea5ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.238+0000 7f8b090f0640 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5ea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.238+0000 7f8b090f0640 1 -- 172.21.15.77:0/4283881672 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dea5034b0 con 0x560dea5e2170 2024-03-16T09:44:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.238+0000 7f8b01ffb640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8af40a2a00 0x7f8af40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.238+0000 7f8b03fff640 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.238+0000 7f8b01ffb640 1 -- 172.21.15.77:0/1091551258 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8af00cde50 con 0x7f8b04102b30 2024-03-16T09:44:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.238+0000 7f8b090f0640 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5e2590 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009df0 tx=0x7f8afc032610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ea223267302cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.238+0000 7f8b088ef640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8af40a2a00 0x7f8af40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.239+0000 7f8aeeffd640 1 -- 172.21.15.77:0/4283881672 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc041020 con 0x560dea5e2170 2024-03-16T09:44:38.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.239+0000 7f8aeeffd640 1 -- 172.21.15.77:0/4283881672 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8afc003e20 con 0x560dea5e2170 2024-03-16T09:44:38.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.239+0000 7f8aeeffd640 1 -- 172.21.15.77:0/4283881672 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc039040 con 0x560dea5e2170 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 -- 172.21.15.77:0/4283881672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 msgr2=0x560dea5e2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5e2590 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009df0 tx=0x7f8afc032610 comp rx=0 tx=0).stop 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b088ef640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8af40a2a00 0x7f8af40a4ec0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x560dea5e2f60 tx=0x7f8af8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 -- 172.21.15.77:0/4283881672 shutdown_connections 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 0x560dea5e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/4283881672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5e2590 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 -- 172.21.15.77:0/4283881672 >> 172.21.15.77:0/4283881672 conn(0x560dea5dd6e0 msgr2=0x560dea5ddae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 -- 172.21.15.77:0/4283881672 shutdown_connections 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 -- 172.21.15.77:0/4283881672 wait complete. 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.240+0000 7f8b0d81e740 1 -- start start 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 1 Processor -- start 2024-03-16T09:44:38.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 0x560dea5f1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5f8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560dea5ef030 con 0x560dea5e7cd0 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560dea503160 con 0x560dea5e2170 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560dea5ef1b0 con 0x560dea5e2b60 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b0d81e740 10 client.0 _ll_register_callbacks cb 0x560dea4e7150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b090f0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b090f0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5f8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41826/0 (socket says 172.21.15.77:41826) 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.241+0000 7f8b090f0640 1 -- 172.21.15.77:0/1419476995 learned_addr learned my addr 172.21.15.77:0/1419476995 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8b090f0640 1 -- 172.21.15.77:0/1419476995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 msgr2=0x560dea5f1010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8b090f0640 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 0x560dea5f1010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8b090f0640 1 -- 172.21.15.77:0/1419476995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 msgr2=0x560dea5f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8b090f0640 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8b090f0640 1 -- 172.21.15.77:0/1419476995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dea3257d0 con 0x560dea5e7cd0 2024-03-16T09:44:38.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8b090f0640 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5f8050 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8afc02c700 tx=0x7f8afc032dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68aa29a554bae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8ae3fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:38.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8ae3fff640 1 -- 172.21.15.77:0/1419476995 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc04a030 con 0x560dea5e7cd0 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8ae3fff640 1 -- 172.21.15.77:0/1419476995 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8afc003970 con 0x560dea5e7cd0 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.242+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8afc003e20 con 0x560dea5e7cd0 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.243+0000 7f8ae3fff640 1 -- 172.21.15.77:0/1419476995 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8afc039050 con 0x560dea5e7cd0 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.243+0000 7f8b0d81e740 10 client.16176 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.243+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560dea25b3f0 con 0x560dea5e7cd0 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.243+0000 7f8ae3fff640 1 -- 172.21.15.77:0/1419476995 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8afc003b70 con 0x560dea5e7cd0 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.244+0000 7f8ae3fff640 1 -- 172.21.15.77:0/1419476995 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8afc03c030 con 0x560dea5e7cd0 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.244+0000 7f8b0d81e740 10 client.16176 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.244+0000 7f8b0d81e740 10 client.16176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.244+0000 7f8b0d81e740 20 client.16176 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.244+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea5fc1b0 0x560dea5fe5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.245+0000 7f8b0d81e740 4 client.16176 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:38.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.245+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8afc003e20 con 0x560dea5fc1b0 2024-03-16T09:44:38.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.246+0000 7f8b088ef640 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea5fc1b0 0x560dea5fe5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.247+0000 7f8b088ef640 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea5fc1b0 0x560dea5fe5a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6271774f04db970f server_cookie=450bbf79f86214b0 in_seq=0 out_seq=0 2024-03-16T09:44:38.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.247+0000 7f8ae3fff640 10 client.16176 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.248+0000 7f8ae3fff640 1 -- 172.21.15.77:0/1419476995 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f8afc003e20 con 0x560dea5fc1b0 2024-03-16T09:44:38.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.248+0000 7f8ae3fff640 10 client.16176 handle_command_reply: tid=0 2024-03-16T09:44:38.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.248+0000 7f8b0d81e740 1 client.16176 shutdown 2024-03-16T09:44:38.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 msgr2=0x560dea5f8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5f8050 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8afc02c700 tx=0x7f8afc032dd0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 shutdown_connections 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea5fc1b0 0x560dea5fe5a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e7cd0 0x560dea5f8050 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2b60 0x560dea5f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8ae3fff640 0 client.16176 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/1419476995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2170 0x560dea5f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 >> 172.21.15.77:0/1419476995 conn(0x560dea5dd6e0 msgr2=0x560dea5dc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 shutdown_connections 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.249+0000 7f8b0d81e740 1 -- 172.21.15.77:0/1419476995 wait complete. 2024-03-16T09:44:38.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.250+0000 7f8b0d81e740 20 client.16176 trim_cache size 0 max 16384 2024-03-16T09:44:38.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.282+0000 7f8b0d81e740 1 Processor -- start 2024-03-16T09:44:38.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b0d81e740 1 -- start start 2024-03-16T09:44:38.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2170 0x560dea5dc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2b60 0x560dea5f8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e7cd0 0x560dea5dc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b041a9340 con 0x560dea5e2170 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b041a6c10 con 0x560dea5e2b60 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b041a4a80 con 0x560dea5e7cd0 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b088ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2170 0x560dea5dc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b088ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2170 0x560dea5dc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41838/0 (socket says 172.21.15.77:41838) 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b088ef640 1 -- 172.21.15.77:0/3530896756 learned_addr learned my addr 172.21.15.77:0/3530896756 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.283+0000 7f8b03fff640 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2b60 0x560dea5f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8b088ef640 1 -- 172.21.15.77:0/3530896756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e7cd0 msgr2=0x560dea5dc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8b088ef640 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e7cd0 0x560dea5dc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8b088ef640 1 -- 172.21.15.77:0/3530896756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2b60 msgr2=0x560dea5f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8b088ef640 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2b60 0x560dea5f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8b088ef640 1 -- 172.21.15.77:0/3530896756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dea21b0f0 con 0x560dea5e2170 2024-03-16T09:44:38.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8b088ef640 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2170 0x560dea5dc070 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x560dea5e35d0 tx=0x7f8af80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b7b659b56b19e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8aee7fc640 1 -- 172.21.15.77:0/3530896756 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8015ca0 con 0x560dea5e2170 2024-03-16T09:44:38.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8aee7fc640 1 -- 172.21.15.77:0/3530896756 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af8029030 con 0x560dea5e2170 2024-03-16T09:44:38.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.284+0000 7f8aee7fc640 1 -- 172.21.15.77:0/3530896756 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8021a10 con 0x560dea5e2170 2024-03-16T09:44:38.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 -- 172.21.15.77:0/3530896756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2170 msgr2=0x560dea5dc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2170 0x560dea5dc070 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x560dea5e35d0 tx=0x7f8af80047d0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 -- 172.21.15.77:0/3530896756 shutdown_connections 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e7cd0 0x560dea5dc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e2b60 0x560dea5f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/3530896756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2170 0x560dea5dc070 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 -- 172.21.15.77:0/3530896756 >> 172.21.15.77:0/3530896756 conn(0x560dea5dd6e0 msgr2=0x560dea5e40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 -- 172.21.15.77:0/3530896756 shutdown_connections 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.285+0000 7f8b0d81e740 1 -- 172.21.15.77:0/3530896756 wait complete. 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 -- start start 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 Processor -- start 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2170 0x560dea5f5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2b60 0x560dea5f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e7cd0 0x560dea5f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b041a50d0 con 0x560dea5e2b60 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b041a6540 con 0x560dea5e7cd0 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b041a57a0 con 0x560dea5e2170 2024-03-16T09:44:38.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b0d81e740 10 client.0 _ll_register_callbacks cb 0x7f8af00d18e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2b60 0x560dea5f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.286+0000 7f8b03fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2b60 0x560dea5f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41844/0 (socket says 172.21.15.77:41844) 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b03fff640 1 -- 172.21.15.77:0/856254136 learned_addr learned my addr 172.21.15.77:0/856254136 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b090f0640 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e7cd0 0x560dea5f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b03fff640 1 -- 172.21.15.77:0/856254136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2170 msgr2=0x560dea5f5640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b03fff640 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2170 0x560dea5f5640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b03fff640 1 -- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e7cd0 msgr2=0x560dea5f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b03fff640 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e7cd0 0x560dea5f6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b03fff640 1 -- 172.21.15.77:0/856254136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dea50e9d0 con 0x560dea5e2b60 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8b03fff640 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2b60 0x560dea5f5b80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8af0096c60 tx=0x7f8af009a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acf579d79c6a1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8ae3fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.287+0000 7f8ae3fff640 1 -- 172.21.15.77:0/856254136 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af001e040 con 0x560dea5e2b60 2024-03-16T09:44:38.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.288+0000 7f8ae3fff640 1 -- 172.21.15.77:0/856254136 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8af0096e70 con 0x560dea5e2b60 2024-03-16T09:44:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.288+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8af8029030 con 0x560dea5e2b60 2024-03-16T09:44:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.288+0000 7f8ae3fff640 1 -- 172.21.15.77:0/856254136 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8af00d1500 con 0x560dea5e2b60 2024-03-16T09:44:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.288+0000 7f8b0d81e740 10 client.16182 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.288+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560dea5f8290 con 0x560dea5e2b60 2024-03-16T09:44:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.288+0000 7f8ae3fff640 1 -- 172.21.15.77:0/856254136 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af8029030 con 0x560dea5e2b60 2024-03-16T09:44:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.288+0000 7f8ae3fff640 1 -- 172.21.15.77:0/856254136 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8af0085070 con 0x560dea5e2b60 2024-03-16T09:44:38.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.289+0000 7f8b0d81e740 10 client.16182 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.289+0000 7f8b0d81e740 10 client.16182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.289+0000 7f8b0d81e740 20 client.16182 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.289+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea662ac0 0x560dea664ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.289+0000 7f8b0d81e740 4 client.16182 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.289+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560dea554a40 con 0x560dea662ac0 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.289+0000 7f8b088ef640 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea662ac0 0x560dea664ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.291+0000 7f8ae3fff640 10 client.16182 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.291+0000 7f8b088ef640 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea662ac0 0x560dea664ee0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed5f4bfa617c341 server_cookie=16c83c1f0ca5d8b in_seq=0 out_seq=0 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8ae3fff640 1 -- 172.21.15.77:0/856254136 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560dea554a40 con 0x560dea662ac0 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8ae3fff640 10 client.16182 handle_command_reply: tid=0 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 client.16182 shutdown 2024-03-16T09:44:38.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2b60 msgr2=0x560dea5f5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2b60 0x560dea5f5b80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8af0096c60 tx=0x7f8af009a690 comp rx=0 tx=0).stop 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 shutdown_connections 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x560dea662ac0 0x560dea664ee0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8ae3fff640 0 client.16182 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560dea5e7cd0 0x560dea5f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560dea5e2b60 0x560dea5f5b80 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 --2- 172.21.15.77:0/856254136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dea5e2170 0x560dea5f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.292+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 >> 172.21.15.77:0/856254136 conn(0x560dea5dd6e0 msgr2=0x560dea5e0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.293+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 shutdown_connections 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.293+0000 7f8b0d81e740 1 -- 172.21.15.77:0/856254136 wait complete. 2024-03-16T09:44:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.293+0000 7f8b0d81e740 20 client.16182 trim_cache size 0 max 16384 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: "65f194f6-dcd7-45d1-8d40-1be2a908656a": { 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "65f194f6-dcd7-45d1-8d40-1be2a908656a", 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:44:38.299 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8af40a2a00 msgr2=0x7f8af40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8af40a2a00 0x7f8af40a4ec0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x560dea5e2f60 tx=0x7f8af8008040 comp rx=0 tx=0).stop 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b04102b30 msgr2=0x7f8b04198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b04102b30 0x7f8b04198050 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8af000b4a0 tx=0x7f8af000b960 comp rx=0 tx=0).stop 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 shutdown_connections 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f8af40a2a00 0x7f8af40a4ec0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b041034f0 0x7f8b0419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b04102b30 0x7f8b04198050 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 --2- 172.21.15.77:0/1091551258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b040ffac0 0x7f8b04197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:38.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.294+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 >> 172.21.15.77:0/1091551258 conn(0x7f8b040fb3d0 msgr2=0x7f8b04105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:38.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.295+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 shutdown_connections 2024-03-16T09:44:38.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:44:38.295+0000 7f8b0ab53640 1 -- 172.21.15.77:0/1091551258 wait complete. 2024-03-16T09:44:38.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:38 smithi129 ceph-mon[35841]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 630 MiB/s wr, 165 op/s 2024-03-16T09:44:38.316 INFO:tasks.cephfs.filesystem:scrub status for tag:65f194f6-dcd7-45d1-8d40-1be2a908656a - {'path': '/', 'tag': '65f194f6-dcd7-45d1-8d40-1be2a908656a', 'options': 'recursive,force'} 2024-03-16T09:44:38.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:38 smithi119 ceph-mon[35632]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 630 MiB/s wr, 165 op/s 2024-03-16T09:44:39.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:39 smithi119 ceph-mon[35632]: scrub queued for path: / 2024-03-16T09:44:39.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:39 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:39.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:39 smithi119 ceph-mon[35632]: scrub summary: active paths [/] 2024-03-16T09:44:39.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:39 smithi119 ceph-mon[35632]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:39 smithi077 ceph-mon[32171]: scrub queued for path: / 2024-03-16T09:44:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:39 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:39 smithi077 ceph-mon[32171]: scrub summary: active paths [/] 2024-03-16T09:44:39.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:39 smithi077 ceph-mon[32171]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:39 smithi129 ceph-mon[35841]: scrub queued for path: / 2024-03-16T09:44:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:39 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:39 smithi129 ceph-mon[35841]: scrub summary: active paths [/] 2024-03-16T09:44:39.551 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:39 smithi129 ceph-mon[35841]: scrub summary: idle+waiting paths [/] 2024-03-16T09:44:40.320 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:40.320 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 rewriters = 746247.62 kB/sec 2024-03-16T09:44:40.320 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 rewriters = 746238.51 kB/sec 2024-03-16T09:44:40.320 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 746247.62 kB/sec 2024-03-16T09:44:40.321 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 746247.62 kB/sec 2024-03-16T09:44:40.321 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 746247.62 kB/sec 2024-03-16T09:44:40.321 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 10485760.00 kB 2024-03-16T09:44:40.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:40 smithi119 ceph-mon[35632]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 639 MiB/s wr, 167 op/s 2024-03-16T09:44:40.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:40 smithi077 ceph-mon[32171]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 639 MiB/s wr, 167 op/s 2024-03-16T09:44:40.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:40 smithi129 ceph-mon[35841]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 639 MiB/s wr, 167 op/s 2024-03-16T09:44:41.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:41 smithi119 ceph-mon[35632]: scrub summary: idle 2024-03-16T09:44:41.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:41 smithi077 ceph-mon[32171]: scrub summary: idle 2024-03-16T09:44:41.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:41 smithi129 ceph-mon[35841]: scrub summary: idle 2024-03-16T09:44:42.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:42 smithi119 ceph-mon[35632]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 738 MiB/s wr, 194 op/s 2024-03-16T09:44:42.509 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:42 smithi077 ceph-mon[32171]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 738 MiB/s wr, 194 op/s 2024-03-16T09:44:42.550 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:42 smithi129 ceph-mon[35841]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 738 MiB/s wr, 194 op/s 2024-03-16T09:44:45.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:44 smithi119 ceph-mon[35632]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 637 MiB/s wr, 168 op/s 2024-03-16T09:44:45.252 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:44 smithi129 ceph-mon[35841]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 637 MiB/s wr, 168 op/s 2024-03-16T09:44:45.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:44 smithi077 ceph-mon[32171]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 637 MiB/s wr, 168 op/s 2024-03-16T09:44:45.389 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:45.389 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 readers = 7891358.50 kB/sec 2024-03-16T09:44:45.389 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 readers = 7889932.95 kB/sec 2024-03-16T09:44:45.389 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 7891358.50 kB/sec 2024-03-16T09:44:45.389 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 7891358.50 kB/sec 2024-03-16T09:44:45.389 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 7891358.50 kB/sec 2024-03-16T09:44:45.389 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 10485760.00 kB 2024-03-16T09:44:46.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- 172.21.15.77:0/3863812301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8168101830 msgr2=0x7f8168101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 --2- 172.21.15.77:0/3863812301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8168101830 0x7f8168101c50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f815c009900 tx=0x7f815c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- 172.21.15.77:0/3863812301 shutdown_connections 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 --2- 172.21.15.77:0/3863812301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81681033e0 0x7f8168109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 --2- 172.21.15.77:0/3863812301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8168102a20 0x7f8168102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 --2- 172.21.15.77:0/3863812301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8168101830 0x7f8168101c50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- 172.21.15.77:0/3863812301 >> 172.21.15.77:0/3863812301 conn(0x7f81680fcda0 msgr2=0x7f81680ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- 172.21.15.77:0/3863812301 shutdown_connections 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- 172.21.15.77:0/3863812301 wait complete. 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 Processor -- start 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- start start 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8168101830 0x7f81681979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8168102a20 0x7f8168197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81681033e0 0x7f816819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f816810d360 con 0x7f81681033e0 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f816810d1e0 con 0x7f8168102a20 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.554+0000 7f816f15b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f816810d4e0 con 0x7f8168101830 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816d6f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81681033e0 0x7f816819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816cef7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8168101830 0x7f81681979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f8167fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8168102a20 0x7f8168197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816d6f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81681033e0 0x7f816819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44204/0 (socket says 172.21.15.77:44204) 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816d6f8640 1 -- 172.21.15.77:0/3384039991 learned_addr learned my addr 172.21.15.77:0/3384039991 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816cef7640 1 -- 172.21.15.77:0/3384039991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8168102a20 msgr2=0x7f8168197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816cef7640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8168102a20 0x7f8168197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816cef7640 1 -- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81681033e0 msgr2=0x7f816819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816cef7640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81681033e0 0x7f816819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816cef7640 1 -- 172.21.15.77:0/3384039991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816819f5c0 con 0x7f8168101830 2024-03-16T09:44:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816d6f8640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81681033e0 0x7f816819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816cef7640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8168101830 0x7f81681979a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f815c036040 tx=0x7f815c005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f8165ffb640 1 -- 172.21.15.77:0/3384039991 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c043020 con 0x7f8168101830 2024-03-16T09:44:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f816819f850 con 0x7f8168101830 2024-03-16T09:44:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f8165ffb640 1 -- 172.21.15.77:0/3384039991 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f815c004350 con 0x7f8168101830 2024-03-16T09:44:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.555+0000 7f8165ffb640 1 -- 172.21.15.77:0/3384039991 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c003ab0 con 0x7f8168101830 2024-03-16T09:44:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.556+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f816819fdc0 con 0x7f8168101830 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.558+0000 7f8165ffb640 1 -- 172.21.15.77:0/3384039991 <== mon.1 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f815c046020 con 0x7f8168101830 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.558+0000 7f8171e27740 1 Processor -- start 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.558+0000 7f8171e27740 1 -- start start 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.558+0000 7f8165ffb640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f81580a28a0 0x7f81580a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8171e27740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587c340 0x55fda587c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8171e27740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587cd30 0x55fda5881830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8171e27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda5884330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8171e27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fda57e1a40 con 0x55fda5881ea0 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8171e27740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fda578a920 con 0x55fda587c340 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8171e27740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fda57ddae0 con 0x55fda587cd30 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816d6f8640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f81580a28a0 0x7f81580a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816d6f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda5884330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8167fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587cd30 0x55fda5881830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8165ffb640 1 -- 172.21.15.77:0/3384039991 <== mon.1 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f815c0f0b90 con 0x7f8168101830 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816cef7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587c340 0x55fda587c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8167fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587cd30 0x55fda5881830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:38164/0 (socket says 172.21.15.77:38164) 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8167fff640 1 -- 172.21.15.77:0/3621395740 learned_addr learned my addr 172.21.15.77:0/3621395740 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816cef7640 1 -- 172.21.15.77:0/3621395740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587cd30 msgr2=0x55fda5881830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816cef7640 1 --2- 172.21.15.77:0/3621395740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587cd30 0x55fda5881830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816cef7640 1 -- 172.21.15.77:0/3621395740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 msgr2=0x55fda5884330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816cef7640 1 --2- 172.21.15.77:0/3621395740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda5884330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816cef7640 1 -- 172.21.15.77:0/3621395740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda57a2460 con 0x55fda587c340 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816d6f8640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f81580a28a0 0x7f81580a4d60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8160009fd0 tx=0x7f8160009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f816cef7640 1 --2- 172.21.15.77:0/3621395740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587c340 0x55fda587c760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f815c0bd470 tx=0x7f815c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfeae444acb799c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.559+0000 7f8152ffd640 1 -- 172.21.15.77:0/3621395740 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c0d6780 con 0x55fda587c340 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8152ffd640 1 -- 172.21.15.77:0/3621395740 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f815c0db170 con 0x55fda587c340 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8152ffd640 1 -- 172.21.15.77:0/3621395740 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c0d65a0 con 0x55fda587c340 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 -- 172.21.15.77:0/3621395740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587c340 msgr2=0x55fda587c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 --2- 172.21.15.77:0/3621395740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587c340 0x55fda587c760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f815c0bd470 tx=0x7f815c02bc30 comp rx=0 tx=0).stop 2024-03-16T09:44:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 -- 172.21.15.77:0/3621395740 shutdown_connections 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 --2- 172.21.15.77:0/3621395740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda5884330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 --2- 172.21.15.77:0/3621395740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587cd30 0x55fda5881830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 --2- 172.21.15.77:0/3621395740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587c340 0x55fda587c760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 -- 172.21.15.77:0/3621395740 >> 172.21.15.77:0/3621395740 conn(0x55fda58778b0 msgr2=0x55fda5877cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 -- 172.21.15.77:0/3621395740 shutdown_connections 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 -- 172.21.15.77:0/3621395740 wait complete. 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.560+0000 7f8171e27740 1 -- start start 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 1 Processor -- start 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda5886400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587cd30 0x55fda5886940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda588d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fda5884c10 con 0x55fda5881ea0 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fda579dd70 con 0x55fda587cd30 2024-03-16T09:44:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fda5884d90 con 0x55fda587c340 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8171e27740 10 client.0 _ll_register_callbacks cb 0x55fda54f5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda588d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda588d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44232/0 (socket says 172.21.15.77:44232) 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816cef7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda5886400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 -- 172.21.15.77:0/1303011986 learned_addr learned my addr 172.21.15.77:0/1303011986 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f8167fff640 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587cd30 0x55fda5886940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 -- 172.21.15.77:0/1303011986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 msgr2=0x55fda5886400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda5886400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 -- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587cd30 msgr2=0x55fda5886940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587cd30 0x55fda5886940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816d6f8640 1 -- 172.21.15.77:0/1303011986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda54f6210 con 0x55fda5881ea0 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.561+0000 7f816cef7640 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda5886400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f816d6f8640 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda588d980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8160010c40 tx=0x7f816001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9645d67a2cf31499 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8147fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8147fff640 1 -- 172.21.15.77:0/1303011986 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160004b20 con 0x55fda5881ea0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8147fff640 1 -- 172.21.15.77:0/1303011986 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8160009040 con 0x55fda5881ea0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fda57eef80 con 0x55fda5881ea0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8147fff640 1 -- 172.21.15.77:0/1303011986 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8160004cc0 con 0x55fda5881ea0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 10 client.16194 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fda55c2620 con 0x55fda5881ea0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8147fff640 1 -- 172.21.15.77:0/1303011986 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fda57eef80 con 0x55fda5881ea0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8147fff640 1 -- 172.21.15.77:0/1303011986 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8160020070 con 0x55fda5881ea0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 10 client.16194 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 20 client.16194 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda5892260 0x55fda58946f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 4 client.16194 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.562+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fda57e92d0 con 0x55fda5892260 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.563+0000 7f816cef7640 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda5892260 0x55fda58946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.563+0000 7f8147fff640 10 client.16194 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.563+0000 7f816cef7640 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda5892260 0x55fda58946f0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c93c641a12b19c6c server_cookie=ce43754da489f4f7 in_seq=0 out_seq=0 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8147fff640 1 -- 172.21.15.77:0/1303011986 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fda57e92d0 con 0x55fda5892260 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8147fff640 10 client.16194 handle_command_reply: tid=0 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 client.16194 shutdown 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 msgr2=0x55fda588d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda588d980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8160010c40 tx=0x7f816001d000 comp rx=0 tx=0).stop 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 shutdown_connections 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda5892260 0x55fda58946f0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda5881ea0 0x55fda588d980 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda587cd30 0x55fda5886940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 --2- 172.21.15.77:0/1303011986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda5886400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8147fff640 0 client.16194 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 >> 172.21.15.77:0/1303011986 conn(0x55fda58778b0 msgr2=0x55fda5876bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 shutdown_connections 2024-03-16T09:44:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 1 -- 172.21.15.77:0/1303011986 wait complete. 2024-03-16T09:44:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.564+0000 7f8171e27740 20 client.16194 trim_cache size 0 max 16384 2024-03-16T09:44:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 Processor -- start 2024-03-16T09:44:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 -- start start 2024-03-16T09:44:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda587e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda587e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f81681a91f0 con 0x55fda587cd30 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f81681a6ac0 con 0x55fda5881ea0 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8171e27740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f81681a4910 con 0x55fda587c340 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda587e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda587e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44242/0 (socket says 172.21.15.77:44242) 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 -- 172.21.15.77:0/2563798223 learned_addr learned my addr 172.21.15.77:0/2563798223 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f816cef7640 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda587e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f816d6f8640 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 -- 172.21.15.77:0/2563798223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 msgr2=0x55fda587e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda587e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 -- 172.21.15.77:0/2563798223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 msgr2=0x55fda588d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 -- 172.21.15.77:0/2563798223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda5873310 con 0x55fda587cd30 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.579+0000 7f8167fff640 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda587e630 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f81540058c0 tx=0x7f8154015bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254e441255758cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f81527fc640 1 -- 172.21.15.77:0/2563798223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154024070 con 0x55fda587cd30 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f81527fc640 1 -- 172.21.15.77:0/2563798223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8154009d80 con 0x55fda587cd30 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f81527fc640 1 -- 172.21.15.77:0/2563798223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815400eb70 con 0x55fda587cd30 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 -- 172.21.15.77:0/2563798223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 msgr2=0x55fda587e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda587e630 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f81540058c0 tx=0x7f8154015bc0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 -- 172.21.15.77:0/2563798223 shutdown_connections 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda587e630 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 --2- 172.21.15.77:0/2563798223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda587e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 -- 172.21.15.77:0/2563798223 >> 172.21.15.77:0/2563798223 conn(0x55fda58778b0 msgr2=0x55fda5877c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 -- 172.21.15.77:0/2563798223 shutdown_connections 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 -- 172.21.15.77:0/2563798223 wait complete. 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.580+0000 7f8171e27740 1 -- start start 2024-03-16T09:44:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 1 Processor -- start 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda588b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda588b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f81681a4f80 con 0x55fda587cd30 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f81681a63f0 con 0x55fda5881ea0 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f81681a5650 con 0x55fda587c340 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8171e27740 10 client.0 _ll_register_callbacks cb 0x7f8160010930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda588b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda588b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44258/0 (socket says 172.21.15.77:44258) 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 -- 172.21.15.77:0/1764711878 learned_addr learned my addr 172.21.15.77:0/1764711878 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f816cef7640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda588b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f816d6f8640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 -- 172.21.15.77:0/1764711878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 msgr2=0x55fda588b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda588b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 -- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 msgr2=0x55fda588c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f8167fff640 1 -- 172.21.15.77:0/1764711878 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda588cc20 con 0x55fda587cd30 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.581+0000 7f816d6f8640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8167fff640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda588b620 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8154005890 tx=0x7f8154015300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45e8c7f653d5618 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8147fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8147fff640 1 -- 172.21.15.77:0/1764711878 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154024040 con 0x55fda587cd30 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8147fff640 1 -- 172.21.15.77:0/1764711878 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f815400eb90 con 0x55fda587cd30 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8154009d80 con 0x55fda587cd30 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8147fff640 1 -- 172.21.15.77:0/1764711878 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f815400ed90 con 0x55fda587cd30 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 10 client.16200 fetch_fsmap learned FSMap version 21 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fda58f8660 con 0x55fda587cd30 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8147fff640 1 -- 172.21.15.77:0/1764711878 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8154009d80 con 0x55fda587cd30 2024-03-16T09:44:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8147fff640 1 -- 172.21.15.77:0/1764711878 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8154004680 con 0x55fda587cd30 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 10 client.16200 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 20 client.16200 populate_metadata read hostname 'smithi077' 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda58f9060 0x55fda58fb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 4 client.16200 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fda5767770 con 0x55fda58f9060 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.582+0000 7f816cef7640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda58f9060 0x55fda58fb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.583+0000 7f8147fff640 10 client.16200 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.583+0000 7f816cef7640 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda58f9060 0x55fda58fb480 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d4fcb548e1e07c3 server_cookie=13d5f3e6b7a80bc6 in_seq=0 out_seq=0 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8147fff640 1 -- 172.21.15.77:0/1764711878 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fda5767770 con 0x55fda58f9060 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8147fff640 10 client.16200 handle_command_reply: tid=0 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 client.16200 shutdown 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 msgr2=0x55fda588b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda588b620 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8154005890 tx=0x7f8154015300 comp rx=0 tx=0).stop 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 shutdown_connections 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55fda58f9060 0x55fda58fb480 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fda5881ea0 0x55fda588c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fda587cd30 0x55fda588b620 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8147fff640 0 client.16200 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 --2- 172.21.15.77:0/1764711878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fda587c340 0x55fda588b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 >> 172.21.15.77:0/1764711878 conn(0x55fda58778b0 msgr2=0x55fda58863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.589+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 shutdown_connections 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f8171e27740 1 -- 172.21.15.77:0/1764711878 wait complete. 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f8171e27740 20 client.16200 trim_cache size 0 max 16384 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T09:44:46.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T09:44:46.594 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f81580a28a0 msgr2=0x7f81580a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.594 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f81580a28a0 0x7f81580a4d60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8160009fd0 tx=0x7f8160009f00 comp rx=0 tx=0).stop 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8168101830 msgr2=0x7f81681979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8168101830 0x7f81681979a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f815c036040 tx=0x7f815c005dc0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 shutdown_connections 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f81580a28a0 0x7f81580a4d60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81681033e0 0x7f816819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8168102a20 0x7f8168197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 --2- 172.21.15.77:0/3384039991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8168101830 0x7f81681979a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 >> 172.21.15.77:0/3384039991 conn(0x7f81680fcda0 msgr2=0x7f8168105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 shutdown_connections 2024-03-16T09:44:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T09:44:46.590+0000 7f816f15b640 1 -- 172.21.15.77:0/3384039991 wait complete. 2024-03-16T09:44:47.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:46 smithi119 ceph-mon[35632]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 555 MiB/s wr, 149 op/s 2024-03-16T09:44:47.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:46 smithi077 ceph-mon[32171]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 555 MiB/s wr, 149 op/s 2024-03-16T09:44:47.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:46 smithi129 ceph-mon[35841]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 555 MiB/s wr, 149 op/s 2024-03-16T09:44:48.879 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:48.879 INFO:tasks.workunit.client.0.smithi077.stdout: Children see throughput for 1 re-readers = 7881517.00 kB/sec 2024-03-16T09:44:48.879 INFO:tasks.workunit.client.0.smithi077.stdout: Parent sees throughput for 1 re-readers = 7879959.75 kB/sec 2024-03-16T09:44:48.879 INFO:tasks.workunit.client.0.smithi077.stdout: Min throughput per process = 7881517.00 kB/sec 2024-03-16T09:44:48.879 INFO:tasks.workunit.client.0.smithi077.stdout: Max throughput per process = 7881517.00 kB/sec 2024-03-16T09:44:48.879 INFO:tasks.workunit.client.0.smithi077.stdout: Avg throughput per process = 7881517.00 kB/sec 2024-03-16T09:44:48.879 INFO:tasks.workunit.client.0.smithi077.stdout: Min xfer = 10485760.00 kB 2024-03-16T09:44:49.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:48 smithi119 ceph-mon[35632]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 439 MiB/s wr, 119 op/s 2024-03-16T09:44:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:48 smithi077 ceph-mon[32171]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 439 MiB/s wr, 119 op/s 2024-03-16T09:44:49.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:48 smithi129 ceph-mon[35841]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 439 MiB/s wr, 119 op/s 2024-03-16T09:44:51.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:50 smithi119 ceph-mon[35632]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 248 MiB/s wr, 69 op/s 2024-03-16T09:44:51.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:50 smithi077 ceph-mon[32171]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 248 MiB/s wr, 69 op/s 2024-03-16T09:44:51.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:50 smithi129 ceph-mon[35841]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 248 MiB/s wr, 69 op/s 2024-03-16T09:44:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:52 smithi119 ceph-mon[35632]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 190 MiB/s wr, 57 op/s 2024-03-16T09:44:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:52 smithi077 ceph-mon[32171]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 190 MiB/s wr, 57 op/s 2024-03-16T09:44:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:52 smithi129 ceph-mon[35841]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 190 MiB/s wr, 57 op/s 2024-03-16T09:44:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:54 smithi119 ceph-mon[35632]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 55 MiB/s wr, 21 op/s 2024-03-16T09:44:55.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:54 smithi077 ceph-mon[32171]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 55 MiB/s wr, 21 op/s 2024-03-16T09:44:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:54 smithi129 ceph-mon[35841]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 55 MiB/s wr, 21 op/s 2024-03-16T09:44:56.015 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:56.015 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:56.015 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T09:44:56.015 INFO:tasks.workunit.client.0.smithi077.stdout:iozone test complete. 2024-03-16T09:44:56.016 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-16T09:44:56.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T09:44:56.057 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-03-16T09:44:56.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-16T09:44:56.370 DEBUG:teuthology.parallel:result is None 2024-03-16T09:44:56.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T09:44:56.407 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T09:44:56.407 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-16T09:44:56.417 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:44:56.455 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-16T09:44:56.464 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-16T09:44:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:56 smithi119 ceph-mon[35632]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8.7 MiB/s wr, 11 op/s 2024-03-16T09:44:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:56 smithi077 ceph-mon[32171]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8.7 MiB/s wr, 11 op/s 2024-03-16T09:44:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:56 smithi129 ceph-mon[35841]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8.7 MiB/s wr, 11 op/s 2024-03-16T09:44:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:44:58 smithi119 ceph-mon[35632]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-16T09:44:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:44:58 smithi077 ceph-mon[32171]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-16T09:44:59.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:44:58 smithi129 ceph-mon[35841]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-16T09:45:01.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:00 smithi119 ceph-mon[35632]: pgmap v439: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 853 B/s wr, 42 op/s 2024-03-16T09:45:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:00 smithi077 ceph-mon[32171]: pgmap v439: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 853 B/s wr, 42 op/s 2024-03-16T09:45:01.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:00 smithi129 ceph-mon[35841]: pgmap v439: 97 pgs: 97 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 853 B/s wr, 42 op/s 2024-03-16T09:45:03.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:02 smithi119 ceph-mon[35632]: pgmap v440: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 155 op/s 2024-03-16T09:45:03.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:02 smithi077 ceph-mon[32171]: pgmap v440: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 155 op/s 2024-03-16T09:45:03.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:02 smithi129 ceph-mon[35841]: pgmap v440: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 155 op/s 2024-03-16T09:45:05.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:04 smithi119 ceph-mon[35632]: pgmap v441: 97 pgs: 97 active+clean; 285 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 214 op/s 2024-03-16T09:45:05.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:04 smithi077 ceph-mon[32171]: pgmap v441: 97 pgs: 97 active+clean; 285 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 214 op/s 2024-03-16T09:45:05.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:04 smithi129 ceph-mon[35841]: pgmap v441: 97 pgs: 97 active+clean; 285 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 214 op/s 2024-03-16T09:45:07.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:06 smithi119 ceph-mon[35632]: pgmap v442: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 222 op/s 2024-03-16T09:45:07.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:06 smithi077 ceph-mon[32171]: pgmap v442: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 222 op/s 2024-03-16T09:45:07.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:06 smithi129 ceph-mon[35841]: pgmap v442: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 222 op/s 2024-03-16T09:45:08.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T09:45:08.317 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T09:45:08.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 -- 172.21.15.77:0/3177682830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4102270 msgr2=0x7f93c4104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 --2- 172.21.15.77:0/3177682830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4102270 0x7f93c4104660 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f93b4009900 tx=0x7f93b402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:08.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 -- 172.21.15.77:0/3177682830 shutdown_connections 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 --2- 172.21.15.77:0/3177682830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93c41074d0 0x7f93c4109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 --2- 172.21.15.77:0/3177682830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93c4104ba0 0x7f93c4106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 --2- 172.21.15.77:0/3177682830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4102270 0x7f93c4104660 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 -- 172.21.15.77:0/3177682830 >> 172.21.15.77:0/3177682830 conn(0x7f93c40fbc80 msgr2=0x7f93c40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 -- 172.21.15.77:0/3177682830 shutdown_connections 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.431+0000 7f93c9868640 1 -- 172.21.15.77:0/3177682830 wait complete. 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 Processor -- start 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 -- start start 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93c4102270 0x7f93c4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4104ba0 0x7f93c4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93c41074d0 0x7f93c419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93c410d350 con 0x7f93c4104ba0 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f93c410d1d0 con 0x7f93c4102270 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c9868640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93c410d4d0 con 0x7f93c41074d0 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4104ba0 0x7f93c4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4104ba0 0x7f93c4198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48582/0 (socket says 172.21.15.77:48582) 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 -- 172.21.15.77:0/2853917789 learned_addr learned my addr 172.21.15.77:0/2853917789 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93c4102270 0x7f93c4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c37fe640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93c41074d0 0x7f93c419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 -- 172.21.15.77:0/2853917789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93c41074d0 msgr2=0x7f93c419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93c41074d0 0x7f93c419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 -- 172.21.15.77:0/2853917789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93c4102270 msgr2=0x7f93c4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93c4102270 0x7f93c4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.432+0000 7f93c27fc640 1 -- 172.21.15.77:0/2853917789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c419f6c0 con 0x7f93c4104ba0 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.433+0000 7f93c27fc640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4104ba0 0x7f93c4198030 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f93b800e9f0 tx=0x7f93b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.433+0000 7f93c8866640 1 -- 172.21.15.77:0/2853917789 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ccc0 con 0x7f93c4104ba0 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.433+0000 7f93c8866640 1 -- 172.21.15.77:0/2853917789 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b80054d0 con 0x7f93c4104ba0 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.433+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c419f9b0 con 0x7f93c4104ba0 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.433+0000 7f93c8866640 1 -- 172.21.15.77:0/2853917789 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b8010430 con 0x7f93c4104ba0 2024-03-16T09:45:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.433+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c419feb0 con 0x7f93c4104ba0 2024-03-16T09:45:08.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93c8866640 1 -- 172.21.15.77:0/2853917789 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f93b8005020 con 0x7f93c4104ba0 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 Processor -- start 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 -- start start 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffbe590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93c8866640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f93ac0a2800 0x7f93ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffc6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f42ff23500 con 0x55f42ffbeb60 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f42fecbbe0 con 0x55f42ffc3cd0 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.435+0000 7f93cc44f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f42ff1f5a0 con 0x55f42ffbe170 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48596/0 (socket says 172.21.15.77:48596) 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 -- 172.21.15.77:0/1694401209 learned_addr learned my addr 172.21.15.77:0/1694401209 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c37fe640 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffc6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f93ac0a2800 0x7f93ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c37fe640 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffbe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c8866640 1 -- 172.21.15.77:0/2853917789 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f93b80cede0 con 0x7f93c4104ba0 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 -- 172.21.15.77:0/1694401209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 msgr2=0x55f42ffbe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffbe590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 -- 172.21.15.77:0/1694401209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 msgr2=0x55f42ffc6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffc6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 -- 172.21.15.77:0/1694401209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f42fedf4b0 con 0x55f42ffbeb60 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c37fe640 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffc6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f93ac0a2800 0x7f93ac0a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f93b4009230 tx=0x7f93b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93c27fc640 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc3660 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f93b806e490 tx=0x7f93b806d410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab1a0e01cb01b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93a9ffb640 1 -- 172.21.15.77:0/1694401209 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b80a0800 con 0x55f42ffbeb60 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93a9ffb640 1 -- 172.21.15.77:0/1694401209 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b80046c0 con 0x55f42ffbeb60 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.436+0000 7f93a9ffb640 1 -- 172.21.15.77:0/1694401209 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ce60 con 0x55f42ffbeb60 2024-03-16T09:45:08.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 -- 172.21.15.77:0/1694401209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 msgr2=0x55f42ffc3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc3660 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f93b806e490 tx=0x7f93b806d410 comp rx=0 tx=0).stop 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 -- 172.21.15.77:0/1694401209 shutdown_connections 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffc6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc3660 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1694401209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffbe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 -- 172.21.15.77:0/1694401209 >> 172.21.15.77:0/1694401209 conn(0x55f42ffb96e0 msgr2=0x55f42ffb9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 -- 172.21.15.77:0/1694401209 shutdown_connections 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 -- 172.21.15.77:0/1694401209 wait complete. 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.437+0000 7f93cc44f740 1 -- start start 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 Processor -- start 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffc86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffcfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f42ffc6c80 con 0x55f42ffbeb60 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f42fedf160 con 0x55f42ffc3cd0 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f42ffc6e00 con 0x55f42ffbe170 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 10 client.0 _ll_register_callbacks cb 0x55f42fec3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48612/0 (socket says 172.21.15.77:48612) 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 -- 172.21.15.77:0/305888196 learned_addr learned my addr 172.21.15.77:0/305888196 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c37fe640 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffcfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffc86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 -- 172.21.15.77:0/305888196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 msgr2=0x55f42ffc86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffc86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 -- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 msgr2=0x55f42ffcfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffcfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 -- 172.21.15.77:0/305888196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f42fd017d0 con 0x55f42ffbeb60 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93c27fc640 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc8c30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f93b809b720 tx=0x7f93b8004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9731dd6438b6d32f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93a2ffd640 1 -- 172.21.15.77:0/305888196 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ce60 con 0x55f42ffbeb60 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93a2ffd640 1 -- 172.21.15.77:0/305888196 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f93b80a0ae0 con 0x55f42ffbeb60 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.438+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f42ff30a40 con 0x55f42ffbeb60 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93a2ffd640 1 -- 172.21.15.77:0/305888196 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f93b8005310 con 0x55f42ffbeb60 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 10 client.16212 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f42fc373f0 con 0x55f42ffbeb60 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93a2ffd640 1 -- 172.21.15.77:0/305888196 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f42ff30a40 con 0x55f42ffbeb60 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93a2ffd640 1 -- 172.21.15.77:0/305888196 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f93b809b9f0 con 0x55f42ffbeb60 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 10 client.16212 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 20 client.16212 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f42ffd3dd0 0x55f42ffd61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 4 client.16212 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:45:08.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f42ff2ad90 con 0x55f42ffd3dd0 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.439+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f42ffd3dd0 0x55f42ffd61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93a2ffd640 10 client.16212 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f42ffd3dd0 0x55f42ffd61c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aebed66aaae53ab8 server_cookie=9b4b6d4c7cbe288 in_seq=0 out_seq=0 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93a2ffd640 1 -- 172.21.15.77:0/305888196 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f42ff2ad90 con 0x55f42ffd3dd0 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93a2ffd640 10 client.16212 handle_command_reply: tid=0 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 client.16212 shutdown 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 msgr2=0x55f42ffc8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc8c30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f93b809b720 tx=0x7f93b8004960 comp rx=0 tx=0).stop 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 shutdown_connections 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f42ffd3dd0 0x55f42ffd61c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f42ffcfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93a2ffd640 0 client.16212 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffc8c30 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 --2- 172.21.15.77:0/305888196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffc86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.440+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 >> 172.21.15.77:0/305888196 conn(0x55f42ffb96e0 msgr2=0x55f42ffb8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.441+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 shutdown_connections 2024-03-16T09:45:08.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.441+0000 7f93cc44f740 1 -- 172.21.15.77:0/305888196 wait complete. 2024-03-16T09:45:08.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.441+0000 7f93cc44f740 20 client.16212 trim_cache size 0 max 16384 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 Processor -- start 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 -- start start 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffbe170 0x55f42ffb48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbeb60 0x55f42ffcfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffc3cd0 0x55f42ffb8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41ab490 con 0x55f42ffc3cd0 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41a8d60 con 0x55f42ffbe170 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93cc44f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41a6bb0 con 0x55f42ffbeb60 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffc3cd0 0x55f42ffb8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffc3cd0 0x55f42ffb8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48626/0 (socket says 172.21.15.77:48626) 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 -- 172.21.15.77:0/3494861077 learned_addr learned my addr 172.21.15.77:0/3494861077 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c27fc640 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbeb60 0x55f42ffcfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffbe170 0x55f42ffb48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 -- 172.21.15.77:0/3494861077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbeb60 msgr2=0x55f42ffcfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbeb60 0x55f42ffcfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 -- 172.21.15.77:0/3494861077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffbe170 msgr2=0x55f42ffb48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffbe170 0x55f42ffb48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.458+0000 7f93c37fe640 1 -- 172.21.15.77:0/3494861077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f42fbf70f0 con 0x55f42ffc3cd0 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93c37fe640 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffc3cd0 0x55f42ffb8070 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f93bc011380 tx=0x7f93bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e256b9ff4e42fb03 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93a97fa640 1 -- 172.21.15.77:0/3494861077 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc011cd0 con 0x55f42ffc3cd0 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93a97fa640 1 -- 172.21.15.77:0/3494861077 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93bc010930 con 0x55f42ffc3cd0 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93a97fa640 1 -- 172.21.15.77:0/3494861077 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc005330 con 0x55f42ffc3cd0 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 -- 172.21.15.77:0/3494861077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffc3cd0 msgr2=0x55f42ffb8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffc3cd0 0x55f42ffb8070 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f93bc011380 tx=0x7f93bc004ab0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 -- 172.21.15.77:0/3494861077 shutdown_connections 2024-03-16T09:45:08.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffc3cd0 0x55f42ffb8070 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbeb60 0x55f42ffcfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 --2- 172.21.15.77:0/3494861077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffbe170 0x55f42ffb48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 -- 172.21.15.77:0/3494861077 >> 172.21.15.77:0/3494861077 conn(0x55f42ffb96e0 msgr2=0x55f42ffc86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 -- 172.21.15.77:0/3494861077 shutdown_connections 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.459+0000 7f93cc44f740 1 -- 172.21.15.77:0/3494861077 wait complete. 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 -- start start 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 Processor -- start 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffcd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f430038010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41a7220 con 0x55f42ffbeb60 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41a8690 con 0x55f42ffc3cd0 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41a78f0 con 0x55f42ffbe170 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93cc44f740 10 client.0 _ll_register_callbacks cb 0x7f93bc009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:08.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffcd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93c27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffcd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48640/0 (socket says 172.21.15.77:48640) 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93c27fc640 1 -- 172.21.15.77:0/1818166746 learned_addr learned my addr 172.21.15.77:0/1818166746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.460+0000 7f93c37fe640 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f430038010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93c27fc640 1 -- 172.21.15.77:0/1818166746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 msgr2=0x55f42ffccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93c27fc640 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93c27fc640 1 -- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 msgr2=0x55f430038010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93c27fc640 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f430038010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93c27fc640 1 -- 172.21.15.77:0/1818166746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f42feea9d0 con 0x55f42ffbeb60 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93c27fc640 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffcd080 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f93b809b490 tx=0x7f93b80961b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d41a05d2a08a39a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93a2ffd640 1 -- 172.21.15.77:0/1818166746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ce60 con 0x55f42ffbeb60 2024-03-16T09:45:08.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93a2ffd640 1 -- 172.21.15.77:0/1818166746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f93b80a3ca0 con 0x55f42ffbeb60 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93bc010930 con 0x55f42ffbeb60 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93a2ffd640 1 -- 172.21.15.77:0/1818166746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f93b80a0850 con 0x55f42ffbeb60 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 10 client.16218 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f430039610 con 0x55f42ffbeb60 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93a2ffd640 1 -- 172.21.15.77:0/1818166746 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93bc010930 con 0x55f42ffbeb60 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93a2ffd640 1 -- 172.21.15.77:0/1818166746 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f93b80afb70 con 0x55f42ffbeb60 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 10 client.16218 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 10 client.16218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 20 client.16218 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f430039d20 0x55f43003c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 4 client.16218 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.461+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f42fea8540 con 0x55f430039d20 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f430039d20 0x55f43003c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93a2ffd640 10 client.16218 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:08.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93c2ffd640 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f430039d20 0x55f43003c140 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a3d4cacb42e1c6 server_cookie=b467e15dce5d056a in_seq=0 out_seq=0 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93a2ffd640 1 -- 172.21.15.77:0/1818166746 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f42fea8540 con 0x55f430039d20 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93a2ffd640 10 client.16218 handle_command_reply: tid=0 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 client.16218 shutdown 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 msgr2=0x55f42ffcd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffcd080 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f93b809b490 tx=0x7f93b80961b0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 shutdown_connections 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x55f430039d20 0x55f43003c140 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f42ffc3cd0 0x55f430038010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f42ffbeb60 0x55f42ffcd080 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93a2ffd640 0 client.16218 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 --2- 172.21.15.77:0/1818166746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f42ffbe170 0x55f42ffccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 >> 172.21.15.77:0/1818166746 conn(0x55f42ffb96e0 msgr2=0x55f42ffbc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 shutdown_connections 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 1 -- 172.21.15.77:0/1818166746 wait complete. 2024-03-16T09:45:08.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.462+0000 7f93cc44f740 20 client.16218 trim_cache size 0 max 16384 2024-03-16T09:45:08.473 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:45:08.473 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T09:45:08.473 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T09:45:08.473 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f93ac0a2800 msgr2=0x7f93ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f93ac0a2800 0x7f93ac0a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f93b4009230 tx=0x7f93b4005fd0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4104ba0 msgr2=0x7f93c4198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4104ba0 0x7f93c4198030 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f93b800e9f0 tx=0x7f93b800eeb0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 shutdown_connections 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f93ac0a2800 0x7f93ac0a4cc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93c41074d0 0x7f93c419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4104ba0 0x7f93c4198030 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 --2- 172.21.15.77:0/2853917789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93c4102270 0x7f93c4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 >> 172.21.15.77:0/2853917789 conn(0x7f93c40fbc80 msgr2=0x7f93c4102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 shutdown_connections 2024-03-16T09:45:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.463+0000 7f93c9868640 1 -- 172.21.15.77:0/2853917789 wait complete. 2024-03-16T09:45:08.477 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T09:45:08.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.575+0000 7f3a397e6640 1 -- 172.21.15.77:0/2979329311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a341096d0 msgr2=0x7f3a3410bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.575+0000 7f3a397e6640 1 --2- 172.21.15.77:0/2979329311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a341096d0 0x7f3a3410bb60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009860 tx=0x7f3a2c02bc00 comp rx=0 tx=0).stop 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.575+0000 7f3a397e6640 1 -- 172.21.15.77:0/2979329311 shutdown_connections 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.575+0000 7f3a397e6640 1 --2- 172.21.15.77:0/2979329311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a341096d0 0x7f3a3410bb60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.575+0000 7f3a397e6640 1 --2- 172.21.15.77:0/2979329311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340741c0 0x7f3a34074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.575+0000 7f3a397e6640 1 --2- 172.21.15.77:0/2979329311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a340757a0 0x7f3a34073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.575+0000 7f3a397e6640 1 -- 172.21.15.77:0/2979329311 >> 172.21.15.77:0/2979329311 conn(0x7f3a340fcd90 msgr2=0x7f3a340ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 -- 172.21.15.77:0/2979329311 shutdown_connections 2024-03-16T09:45:08.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 -- 172.21.15.77:0/2979329311 wait complete. 2024-03-16T09:45:08.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 Processor -- start 2024-03-16T09:45:08.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 -- start start 2024-03-16T09:45:08.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a340741c0 0x7f3a34199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340757a0 0x7f3a3419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a341096d0 0x7f3a341a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3410f4f0 con 0x7f3a340757a0 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3410f370 con 0x7f3a340741c0 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.576+0000 7f3a397e6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3410f670 con 0x7f3a341096d0 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340757a0 0x7f3a3419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a340741c0 0x7f3a34199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340757a0 0x7f3a3419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48656/0 (socket says 172.21.15.77:48656) 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 -- 172.21.15.77:0/3498640686 learned_addr learned my addr 172.21.15.77:0/3498640686 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a337fe640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a341096d0 0x7f3a341a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 -- 172.21.15.77:0/3498640686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a341096d0 msgr2=0x7f3a341a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a341096d0 0x7f3a341a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 -- 172.21.15.77:0/3498640686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a340741c0 msgr2=0x7f3a34199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a340741c0 0x7f3a34199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 -- 172.21.15.77:0/3498640686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a341a17d0 con 0x7f3a340757a0 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a327fc640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340757a0 0x7f3a3419a0f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800e9c0 tx=0x7f3a2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a1bfff640 1 -- 172.21.15.77:0/3498640686 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2800cde0 con 0x7f3a340757a0 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a341a1ac0 con 0x7f3a340757a0 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a1bfff640 1 -- 172.21.15.77:0/3498640686 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a280035f0 con 0x7f3a340757a0 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a1bfff640 1 -- 172.21.15.77:0/3498640686 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28010690 con 0x7f3a340757a0 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.577+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a341a2060 con 0x7f3a340757a0 2024-03-16T09:45:08.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.578+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a34108490 con 0x7f3a340757a0 2024-03-16T09:45:08.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.579+0000 7f3a1bfff640 1 -- 172.21.15.77:0/3498640686 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3a280037a0 con 0x7f3a340757a0 2024-03-16T09:45:08.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.580+0000 7f3a1bfff640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a1c0a2800 0x7f3a1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.580+0000 7f3a32ffd640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a1c0a2800 0x7f3a1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.580+0000 7f3a1bfff640 1 -- 172.21.15.77:0/3498640686 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3a280cee30 con 0x7f3a340757a0 2024-03-16T09:45:08.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.580+0000 7f3a32ffd640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a1c0a2800 0x7f3a1c0a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a24005fd0 tx=0x7f3a24009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.584+0000 7f3a1bfff640 1 -- 172.21.15.77:0/3498640686 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3a2809bf70 con 0x7f3a340757a0 2024-03-16T09:45:08.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.797+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a3402d1b0 con 0x7f3a340757a0 2024-03-16T09:45:08.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.798+0000 7f3a1bfff640 1 -- 172.21.15.77:0/3498640686 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f3a2809b870 con 0x7f3a340757a0 2024-03-16T09:45:08.801 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:08.801 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:45:08.802 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:45:08.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a1c0a2800 msgr2=0x7f3a1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a1c0a2800 0x7f3a1c0a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a24005fd0 tx=0x7f3a24009500 comp rx=0 tx=0).stop 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340757a0 msgr2=0x7f3a3419a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340757a0 0x7f3a3419a0f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800e9c0 tx=0x7f3a2800ee80 comp rx=0 tx=0).stop 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 shutdown_connections 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f3a1c0a2800 0x7f3a1c0a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a341096d0 0x7f3a341a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a340757a0 0x7f3a3419a0f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 --2- 172.21.15.77:0/3498640686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a340741c0 0x7f3a34199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 >> 172.21.15.77:0/3498640686 conn(0x7f3a340fcd90 msgr2=0x7f3a34105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 shutdown_connections 2024-03-16T09:45:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.800+0000 7f3a397e6640 1 -- 172.21.15.77:0/3498640686 wait complete. 2024-03-16T09:45:08.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- 172.21.15.77:0/169002033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b01022b0 msgr2=0x7f76b0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 --2- 172.21.15.77:0/169002033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b01022b0 0x7f76b0108b40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f76a0009900 tx=0x7f76a002bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- 172.21.15.77:0/169002033 shutdown_connections 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 --2- 172.21.15.77:0/169002033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b01022b0 0x7f76b0108b40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 --2- 172.21.15.77:0/169002033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b01018f0 0x7f76b0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 --2- 172.21.15.77:0/169002033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76b0100700 0x7f76b0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- 172.21.15.77:0/169002033 >> 172.21.15.77:0/169002033 conn(0x7f76b00fbc90 msgr2=0x7f76b00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- 172.21.15.77:0/169002033 shutdown_connections 2024-03-16T09:45:08.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- 172.21.15.77:0/169002033 wait complete. 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 Processor -- start 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- start start 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b0100700 0x7f76b0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76b01018f0 0x7f76b019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b01022b0 0x7f76b01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76b010c230 con 0x7f76b0100700 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f76b010c0b0 con 0x7f76b01018f0 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b717e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76b010c3b0 con 0x7f76b01022b0 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b0100700 0x7f76b0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b0100700 0x7f76b0199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48674/0 (socket says 172.21.15.77:48674) 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.917+0000 7f76b4f1a640 1 -- 172.21.15.77:0/1709682047 learned_addr learned my addr 172.21.15.77:0/1709682047 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76affff640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76b01018f0 0x7f76b019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b571b640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b01022b0 0x7f76b01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b4f1a640 1 -- 172.21.15.77:0/1709682047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b01022b0 msgr2=0x7f76b01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b01022b0 0x7f76b01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b4f1a640 1 -- 172.21.15.77:0/1709682047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76b01018f0 msgr2=0x7f76b019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76b01018f0 0x7f76b019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b4f1a640 1 -- 172.21.15.77:0/1709682047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b01a1810 con 0x7f76b0100700 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b0100700 0x7f76b0199bf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f76a800ebc0 tx=0x7f76a800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76adffb640 1 -- 172.21.15.77:0/1709682047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8003b70 con 0x7f76b0100700 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76adffb640 1 -- 172.21.15.77:0/1709682047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76a8003d10 con 0x7f76b0100700 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b01a1b00 con 0x7f76b0100700 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76adffb640 1 -- 172.21.15.77:0/1709682047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8019840 con 0x7f76b0100700 2024-03-16T09:45:08.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.918+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b01a2050 con 0x7f76b0100700 2024-03-16T09:45:08.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76adffb640 1 -- 172.21.15.77:0/1709682047 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f76a80199e0 con 0x7f76b0100700 2024-03-16T09:45:08.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 Processor -- start 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 -- start start 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e87c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e8cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e8f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565010dec490 con 0x565010e8d3b0 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x565010d94c90 con 0x565010e88240 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x565010de8530 con 0x565010e87850 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76adffb640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f769c0a2800 0x7f769c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.921+0000 7f76affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e8f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76b571b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e87c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e8cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e8f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48678/0 (socket says 172.21.15.77:48678) 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76b571b640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f769c0a2800 0x7f769c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 -- 172.21.15.77:0/273324282 learned_addr learned my addr 172.21.15.77:0/273324282 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76adffb640 1 -- 172.21.15.77:0/1709682047 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f76a80cf2c0 con 0x7f76b0100700 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 -- 172.21.15.77:0/273324282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 msgr2=0x565010e87c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e87c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 -- 172.21.15.77:0/273324282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 msgr2=0x565010e8cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e8cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 -- 172.21.15.77:0/273324282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565010e4f500 con 0x565010e8d3b0 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e8cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76b571b640 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e87c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76affff640 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e8f840 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7698009ca0 tx=0x7f7698011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e59c4011580e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76b571b640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f769c0a2800 0x7f769c0a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006010 tx=0x7f76a002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76a6ffd640 1 -- 172.21.15.77:0/273324282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769801e070 con 0x565010e8d3b0 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76a6ffd640 1 -- 172.21.15.77:0/273324282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7698019ae0 con 0x565010e8d3b0 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.922+0000 7f76a6ffd640 1 -- 172.21.15.77:0/273324282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769800e990 con 0x565010e8d3b0 2024-03-16T09:45:08.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 -- 172.21.15.77:0/273324282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 msgr2=0x565010e8f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e8f840 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7698009ca0 tx=0x7f7698011810 comp rx=0 tx=0).stop 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 -- 172.21.15.77:0/273324282 shutdown_connections 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e8f840 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e8cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/273324282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e87c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 -- 172.21.15.77:0/273324282 >> 172.21.15.77:0/273324282 conn(0x565010e82dc0 msgr2=0x565010e831c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 -- 172.21.15.77:0/273324282 shutdown_connections 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 -- 172.21.15.77:0/273324282 wait complete. 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 -- start start 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.923+0000 7f76b9d6b740 1 Processor -- start 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e91d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e88240 0x565010e92260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e8d3b0 0x565010e99260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565010e90360 con 0x565010e87850 2024-03-16T09:45:08.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x565010da8230 con 0x565010e8d3b0 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x565010e904e0 con 0x565010e88240 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 10 client.0 _ll_register_callbacks cb 0x565010de4010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e91d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e91d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48694/0 (socket says 172.21.15.77:48694) 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 -- 172.21.15.77:0/2928927654 learned_addr learned my addr 172.21.15.77:0/2928927654 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b571b640 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e8d3b0 0x565010e99260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76affff640 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e88240 0x565010e92260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 -- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e88240 msgr2=0x565010e92260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e88240 0x565010e92260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 -- 172.21.15.77:0/2928927654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e8d3b0 msgr2=0x565010e99260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e8d3b0 0x565010e99260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 -- 172.21.15.77:0/2928927654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565010db7100 con 0x565010e87850 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e91d20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f76a8098640 tx=0x7f76a806b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446d3ef32ecbd807 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f768ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:08.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f768ffff640 1 -- 172.21.15.77:0/2928927654 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a809c370 con 0x565010e87850 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f768ffff640 1 -- 172.21.15.77:0/2928927654 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f76a80a3f80 con 0x565010e87850 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.924+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7698019ae0 con 0x565010e87850 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f768ffff640 1 -- 172.21.15.77:0/2928927654 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f76a809bc70 con 0x565010e87850 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 10 client.16236 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x565010b003f0 con 0x565010e87850 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f768ffff640 1 -- 172.21.15.77:0/2928927654 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7698019ae0 con 0x565010e87850 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f768ffff640 1 -- 172.21.15.77:0/2928927654 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f76a8086470 con 0x565010e87850 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 10 client.16236 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 10 client.16236 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 20 client.16236 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010e9ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 4 client.16236 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565010df99d0 con 0x565010e9db40 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.925+0000 7f76affff640 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010e9ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f768ffff640 10 client.16236 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76affff640 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010e9ffd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c061a1c012015753 server_cookie=cfb67bed6c2dc0b4 in_seq=0 out_seq=0 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f768ffff640 1 -- 172.21.15.77:0/2928927654 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565010df99d0 con 0x565010e9db40 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f768ffff640 10 client.16236 handle_command_reply: tid=0 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 client.16236 shutdown 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 msgr2=0x565010e91d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e91d20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f76a8098640 tx=0x7f76a806b790 comp rx=0 tx=0).stop 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 shutdown_connections 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010e9ffd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e8d3b0 0x565010e99260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e88240 0x565010e92260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f768ffff640 0 client.16236 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/2928927654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e91d20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.926+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 >> 172.21.15.77:0/2928927654 conn(0x565010e82dc0 msgr2=0x565010e81fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.927+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 shutdown_connections 2024-03-16T09:45:08.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.927+0000 7f76b9d6b740 1 -- 172.21.15.77:0/2928927654 wait complete. 2024-03-16T09:45:08.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.927+0000 7f76b9d6b740 20 client.16236 trim_cache size 0 max 16384 2024-03-16T09:45:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 Processor -- start 2024-03-16T09:45:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 -- start start 2024-03-16T09:45:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e90880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e99220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e8d3b0 0x565010e90dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76b01ad540 con 0x565010e87850 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f76b01aae10 con 0x565010e88240 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76b01a8c60 con 0x565010e8d3b0 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e90880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e90880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48710/0 (socket says 172.21.15.77:48710) 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 -- 172.21.15.77:0/436764072 learned_addr learned my addr 172.21.15.77:0/436764072 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b571b640 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e8d3b0 0x565010e90dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76affff640 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e99220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 -- 172.21.15.77:0/436764072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e8d3b0 msgr2=0x565010e90dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e8d3b0 0x565010e90dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 -- 172.21.15.77:0/436764072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 msgr2=0x565010e99220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e99220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.940+0000 7f76b4f1a640 1 -- 172.21.15.77:0/436764072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565010bca7d0 con 0x565010e87850 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e90880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f76a8099f20 tx=0x7f76a8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91af613cdb76b39d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76a67fc640 1 -- 172.21.15.77:0/436764072 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a809c370 con 0x565010e87850 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76a67fc640 1 -- 172.21.15.77:0/436764072 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76a809fdc0 con 0x565010e87850 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76a67fc640 1 -- 172.21.15.77:0/436764072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a80a41e0 con 0x565010e87850 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 -- 172.21.15.77:0/436764072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 msgr2=0x565010e90880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e90880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f76a8099f20 tx=0x7f76a8000f40 comp rx=0 tx=0).stop 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 -- 172.21.15.77:0/436764072 shutdown_connections 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e8d3b0 0x565010e90dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e99220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/436764072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e87850 0x565010e90880 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 -- 172.21.15.77:0/436764072 >> 172.21.15.77:0/436764072 conn(0x565010e82dc0 msgr2=0x565010e868b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 -- 172.21.15.77:0/436764072 shutdown_connections 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.941+0000 7f76b9d6b740 1 -- 172.21.15.77:0/436764072 wait complete. 2024-03-16T09:45:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 -- start start 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 Processor -- start 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e96af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e97030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e97ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76b01a92d0 con 0x565010e8d3b0 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f76b01aa740 con 0x565010e88240 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76b01a99a0 con 0x565010e87850 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b9d6b740 10 client.0 _ll_register_callbacks cb 0x7f76a800eb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b571b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e97ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b4f1a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e96af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b571b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e97ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48714/0 (socket says 172.21.15.77:48714) 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76b571b640 1 -- 172.21.15.77:0/1142324248 learned_addr learned my addr 172.21.15.77:0/1142324248 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.942+0000 7f76affff640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e97030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b571b640 1 -- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 msgr2=0x565010e96af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b571b640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e96af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b571b640 1 -- 172.21.15.77:0/1142324248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 msgr2=0x565010e97030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b571b640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e97030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b571b640 1 -- 172.21.15.77:0/1142324248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565010e985e0 con 0x565010e8d3b0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e96af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76affff640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e97030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b571b640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e97ff0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f76a0009900 tx=0x7f76a002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca28ecb9e0fafb9f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f768ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f768ffff640 1 -- 172.21.15.77:0/1142324248 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a0056070 con 0x565010e8d3b0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f768ffff640 1 -- 172.21.15.77:0/1142324248 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f76a0004b10 con 0x565010e8d3b0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76a809fdc0 con 0x565010e8d3b0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f768ffff640 1 -- 172.21.15.77:0/1142324248 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f76a0062050 con 0x565010e8d3b0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 10 client.16242 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x565010f04070 con 0x565010e8d3b0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f768ffff640 1 -- 172.21.15.77:0/1142324248 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76a809fdc0 con 0x565010e8d3b0 2024-03-16T09:45:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f768ffff640 1 -- 172.21.15.77:0/1142324248 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f76a004d050 con 0x565010e8d3b0 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 10 client.16242 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 10 client.16242 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 20 client.16242 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010f06ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 4 client.16242 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.943+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 --> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565010df3d20 con 0x565010e9db40 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010f06ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f768ffff640 10 client.16242 ms_handle_connect on v2:172.21.15.129:6838/1514269525 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b4f1a640 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010f06ae0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=df3e11de4b197b4c server_cookie=14426ae757e7b3da in_seq=0 out_seq=0 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f768ffff640 1 -- 172.21.15.77:0/1142324248 <== mds.2 v2:172.21.15.129:6838/1514269525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565010df3d20 con 0x565010e9db40 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f768ffff640 10 client.16242 handle_command_reply: tid=0 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 client.16242 shutdown 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 msgr2=0x565010e97ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e97ff0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f76a0009900 tx=0x7f76a002bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 shutdown_connections 2024-03-16T09:45:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565010e8d3b0 0x565010e97ff0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:6838/1514269525,v1:172.21.15.129:6839/1514269525] conn(0x565010e9db40 0x565010f06ae0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565010e88240 0x565010e97030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f768ffff640 0 client.16242 ms_handle_reset on v2:172.21.15.129:6838/1514269525 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 --2- 172.21.15.77:0/1142324248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x565010e87850 0x565010e96af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 >> 172.21.15.77:0/1142324248 conn(0x565010e82dc0 msgr2=0x565010e81750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 shutdown_connections 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 1 -- 172.21.15.77:0/1142324248 wait complete. 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.944+0000 7f76b9d6b740 20 client.16242 trim_cache size 0 max 16384 2024-03-16T09:45:08.954 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:45:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f769c0a2800 msgr2=0x7f769c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f769c0a2800 0x7f769c0a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006010 tx=0x7f76a002c7e0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b0100700 msgr2=0x7f76b0199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b0100700 0x7f76b0199bf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f76a800ebc0 tx=0x7f76a800c6a0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 shutdown_connections 2024-03-16T09:45:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f769c0a2800 0x7f769c0a4cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b01022b0 0x7f76b01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76b01018f0 0x7f76b019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 --2- 172.21.15.77:0/1709682047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b0100700 0x7f76b0199bf0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 >> 172.21.15.77:0/1709682047 conn(0x7f76b00fbc90 msgr2=0x7f76b0069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 shutdown_connections 2024-03-16T09:45:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:08.945+0000 7f76b717e640 1 -- 172.21.15.77:0/1709682047 wait complete. 2024-03-16T09:45:08.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T09:45:09.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 -- 172.21.15.77:0/2144902112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c8100700 msgr2=0x7f06c8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 --2- 172.21.15.77:0/2144902112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c8100700 0x7f06c8100b20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009ca0 tx=0x7f06c402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 -- 172.21.15.77:0/2144902112 shutdown_connections 2024-03-16T09:45:09.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 --2- 172.21.15.77:0/2144902112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06c81022b0 0x7f06c8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 --2- 172.21.15.77:0/2144902112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c81018f0 0x7f06c8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 --2- 172.21.15.77:0/2144902112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c8100700 0x7f06c8100b20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 -- 172.21.15.77:0/2144902112 >> 172.21.15.77:0/2144902112 conn(0x7f06c80fbc90 msgr2=0x7f06c80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 -- 172.21.15.77:0/2144902112 shutdown_connections 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.060+0000 7f06d0169640 1 -- 172.21.15.77:0/2144902112 wait complete. 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 Processor -- start 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 -- start start 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06c8100700 0x7f06c8196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c81018f0 0x7f06c8196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c81022b0 0x7f06c819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06c810c230 con 0x7f06c81018f0 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f06c810c0b0 con 0x7f06c81022b0 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06c810c3b0 con 0x7f06c8100700 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c81018f0 0x7f06c8196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c81018f0 0x7f06c8196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48730/0 (socket says 172.21.15.77:48730) 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 -- 172.21.15.77:0/2368478406 learned_addr learned my addr 172.21.15.77:0/2368478406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cdf05640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06c8100700 0x7f06c8196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06ce706640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c81022b0 0x7f06c819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 -- 172.21.15.77:0/2368478406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06c8100700 msgr2=0x7f06c8196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06c8100700 0x7f06c8196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 -- 172.21.15.77:0/2368478406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c81022b0 msgr2=0x7f06c819df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c81022b0 0x7f06c819df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 -- 172.21.15.77:0/2368478406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c819e580 con 0x7f06c81018f0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06ce706640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c81022b0 0x7f06c819df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06cd704640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c81018f0 0x7f06c8196ea0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f06b800eb00 tx=0x7f06b8009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06beffd640 1 -- 172.21.15.77:0/2368478406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b801c070 con 0x7f06c81018f0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06beffd640 1 -- 172.21.15.77:0/2368478406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06b800bc30 con 0x7f06c81018f0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.061+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c8069e10 con 0x7f06c81018f0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.062+0000 7f06beffd640 1 -- 172.21.15.77:0/2368478406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8016610 con 0x7f06c81018f0 2024-03-16T09:45:09.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.062+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c806a360 con 0x7f06c81018f0 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06beffd640 1 -- 172.21.15.77:0/2368478406 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f06b801f070 con 0x7f06c81018f0 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 Processor -- start 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 -- start start 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4c850 0x56536ab4cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab51d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab54840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56536aab1490 con 0x56536ab523b0 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56536aa59c90 con 0x56536ab4c850 2024-03-16T09:45:09.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06d2e32740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56536aaad530 con 0x56536ab4d240 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06cd704640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab54840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06ce706640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4c850 0x56536ab4cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06cd704640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab54840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48742/0 (socket says 172.21.15.77:48742) 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06cd704640 1 -- 172.21.15.77:0/1110272604 learned_addr learned my addr 172.21.15.77:0/1110272604 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06cdf05640 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab51d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.064+0000 7f06beffd640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f06b40a2800 0x7f06b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06ce706640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f06b40a2800 0x7f06b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06cdf05640 1 -- 172.21.15.77:0/1110272604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4c850 msgr2=0x56536ab4cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06cdf05640 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4c850 0x56536ab4cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06cdf05640 1 -- 172.21.15.77:0/1110272604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 msgr2=0x56536ab54840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06cdf05640 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab54840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06cdf05640 1 -- 172.21.15.77:0/1110272604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56536ab14500 con 0x56536ab4d240 2024-03-16T09:45:09.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06beffd640 1 -- 172.21.15.77:0/2368478406 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f06b80d0020 con 0x7f06c81018f0 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06cdf05640 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab51d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f06c4037040 tx=0x7f06c4005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02d75473029fd6d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06ce706640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f06b40a2800 0x7f06b40a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x56536ab4daa0 tx=0x7f06c0004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06b3fff640 1 -- 172.21.15.77:0/1110272604 <== mon.1 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4043070 con 0x56536ab4d240 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06b3fff640 1 -- 172.21.15.77:0/1110272604 <== mon.1 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c4032db0 con 0x56536ab4d240 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.065+0000 7f06b3fff640 1 -- 172.21.15.77:0/1110272604 <== mon.1 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4039040 con 0x56536ab4d240 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 -- 172.21.15.77:0/1110272604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 msgr2=0x56536ab51d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab51d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f06c4037040 tx=0x7f06c4005f70 comp rx=0 tx=0).stop 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 -- 172.21.15.77:0/1110272604 shutdown_connections 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab54840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab51d40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1110272604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4c850 0x56536ab4cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 -- 172.21.15.77:0/1110272604 >> 172.21.15.77:0/1110272604 conn(0x56536ab47dc0 msgr2=0x56536ab481c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 -- 172.21.15.77:0/1110272604 shutdown_connections 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 -- 172.21.15.77:0/1110272604 wait complete. 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 -- start start 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.066+0000 7f06d2e32740 1 Processor -- start 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab56d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab572d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab523b0 0x56536ab5e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56536ab55300 con 0x56536ab4c850 2024-03-16T09:45:09.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56536aa6d230 con 0x56536ab4d240 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56536ab55480 con 0x56536ab523b0 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 10 client.0 _ll_register_callbacks cb 0x56536aaa9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab56d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab56d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48754/0 (socket says 172.21.15.77:48754) 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 -- 172.21.15.77:0/4110569780 learned_addr learned my addr 172.21.15.77:0/4110569780 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cd704640 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06ce706640 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab523b0 0x56536ab5e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 -- 172.21.15.77:0/4110569780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab523b0 msgr2=0x56536ab5e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab523b0 0x56536ab5e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 -- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 msgr2=0x56536ab572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 -- 172.21.15.77:0/4110569780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56536aa7c100 con 0x56536ab4c850 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06cdf05640 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab56d90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f06c4032420 tx=0x7f06c4032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbeacd4678ee3e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06b17fa640 1 -- 172.21.15.77:0/4110569780 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c404c030 con 0x56536ab4c850 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06b17fa640 1 -- 172.21.15.77:0/4110569780 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f06c4032800 con 0x56536ab4c850 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.067+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06c4032db0 con 0x56536ab4c850 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06b17fa640 1 -- 172.21.15.77:0/4110569780 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f06c4039070 con 0x56536ab4c850 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 10 client.16254 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56536a7c53f0 con 0x56536ab4c850 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06b17fa640 1 -- 172.21.15.77:0/4110569780 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c4032db0 con 0x56536ab4c850 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06b17fa640 1 -- 172.21.15.77:0/4110569780 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f06c4048570 con 0x56536ab4c850 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 10 client.16254 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 10 client.16254 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 20 client.16254 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536ab62790 0x56536ab64b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 4 client.16254 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56536aabe9d0 con 0x56536ab62790 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.068+0000 7f06cd704640 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536ab62790 0x56536ab64b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06cd704640 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536ab62790 0x56536ab64b80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c33cd1da0efa8fc5 server_cookie=d532e7742d2af05b in_seq=0 out_seq=0 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06b17fa640 10 client.16254 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06b17fa640 1 -- 172.21.15.77:0/4110569780 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56536aabe9d0 con 0x56536ab62790 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06b17fa640 10 client.16254 handle_command_reply: tid=0 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 client.16254 shutdown 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 msgr2=0x56536ab56d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab56d90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f06c4032420 tx=0x7f06c4032fc0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 shutdown_connections 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536ab62790 0x56536ab64b80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab523b0 0x56536ab5e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06b17fa640 0 client.16254 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:45:09.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 --2- 172.21.15.77:0/4110569780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab56d90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 >> 172.21.15.77:0/4110569780 conn(0x56536ab47dc0 msgr2=0x56536ab46fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 shutdown_connections 2024-03-16T09:45:09.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.069+0000 7f06d2e32740 1 -- 172.21.15.77:0/4110569780 wait complete. 2024-03-16T09:45:09.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.070+0000 7f06d2e32740 20 client.16254 trim_cache size 0 max 16384 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 Processor -- start 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 -- start start 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4c850 0x56536ab69330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab5e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab69870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a9210 con 0x56536ab523b0 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a6ae0 con 0x56536ab4d240 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06d2e32740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a4930 con 0x56536ab4c850 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab69870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab69870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48766/0 (socket says 172.21.15.77:48766) 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 -- 172.21.15.77:0/3769372584 learned_addr learned my addr 172.21.15.77:0/3769372584 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06cdf05640 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4c850 0x56536ab69330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06cd704640 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab5e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 -- 172.21.15.77:0/3769372584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4c850 msgr2=0x56536ab69330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4c850 0x56536ab69330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 -- 172.21.15.77:0/3769372584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 msgr2=0x56536ab5e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab5e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 -- 172.21.15.77:0/3769372584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56536a88f7d0 con 0x56536ab523b0 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06ce706640 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab69870 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f06c000f740 tx=0x7f06c0009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca2c12aafe7540e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06b37fe640 1 -- 172.21.15.77:0/3769372584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c0022420 con 0x56536ab523b0 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06b37fe640 1 -- 172.21.15.77:0/3769372584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c0022a70 con 0x56536ab523b0 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.083+0000 7f06b37fe640 1 -- 172.21.15.77:0/3769372584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c001faf0 con 0x56536ab523b0 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 -- 172.21.15.77:0/3769372584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 msgr2=0x56536ab69870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab69870 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f06c000f740 tx=0x7f06c0009710 comp rx=0 tx=0).stop 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 -- 172.21.15.77:0/3769372584 shutdown_connections 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab523b0 0x56536ab69870 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab4d240 0x56536ab5e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 --2- 172.21.15.77:0/3769372584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4c850 0x56536ab69330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 -- 172.21.15.77:0/3769372584 >> 172.21.15.77:0/3769372584 conn(0x56536ab47dc0 msgr2=0x56536ab4b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 -- 172.21.15.77:0/3769372584 shutdown_connections 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 -- 172.21.15.77:0/3769372584 wait complete. 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 -- start start 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.084+0000 7f06d2e32740 1 Processor -- start 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab5d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab5d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab523b0 0x56536abc6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a4fa0 con 0x56536ab4c850 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a6410 con 0x56536ab523b0 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a5670 con 0x56536ab4d240 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 10 client.0 _ll_register_callbacks cb 0x7f06b8098da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab5d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab5d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48772/0 (socket says 172.21.15.77:48772) 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 -- 172.21.15.77:0/1539229935 learned_addr learned my addr 172.21.15.77:0/1539229935 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cd704640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab5d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06ce706640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab523b0 0x56536abc6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 -- 172.21.15.77:0/1539229935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 msgr2=0x56536ab5d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab5d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 -- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab523b0 msgr2=0x56536abc6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab523b0 0x56536abc6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 -- 172.21.15.77:0/1539229935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56536abc7080 con 0x56536ab4c850 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06ce706640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab523b0 0x56536abc6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cd704640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab5d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:45:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06cdf05640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab5d3c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f06c4005df0 tx=0x7f06c4005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c690d997853d9aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06b17fa640 1 -- 172.21.15.77:0/1539229935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4061060 con 0x56536ab4c850 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06b17fa640 1 -- 172.21.15.77:0/1539229935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f06c4048450 con 0x56536ab4c850 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.085+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06c0022a70 con 0x56536ab4c850 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06b17fa640 1 -- 172.21.15.77:0/1539229935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f06c404c040 con 0x56536ab4c850 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 10 client.16260 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56536abc8320 con 0x56536ab4c850 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06b17fa640 1 -- 172.21.15.77:0/1539229935 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c0022a70 con 0x56536ab4c850 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06b17fa640 1 -- 172.21.15.77:0/1539229935 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f06c405ca90 con 0x56536ab4c850 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 10 client.16260 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 10 client.16260 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 20 client.16260 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536abc8c70 0x56536abcb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 4 client.16260 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:45:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 --> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56536aab8d20 con 0x56536abc8c70 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06cd704640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536abc8c70 0x56536abcb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06b17fa640 10 client.16260 ms_handle_connect on v2:172.21.15.119:6836/1948234265 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.086+0000 7f06cd704640 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536abc8c70 0x56536abcb090 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc359ea8f7b3b9cf server_cookie=d49e74400b4dc515 in_seq=0 out_seq=0 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06b17fa640 1 -- 172.21.15.77:0/1539229935 <== mds.1 v2:172.21.15.119:6836/1948234265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56536aab8d20 con 0x56536abc8c70 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06b17fa640 10 client.16260 handle_command_reply: tid=0 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 client.16260 shutdown 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 msgr2=0x56536ab5d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab5d3c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f06c4005df0 tx=0x7f06c4005fb0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 shutdown_connections 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:6836/1948234265,v1:172.21.15.119:6837/1948234265] conn(0x56536abc8c70 0x56536abcb090 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56536ab523b0 0x56536abc6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56536ab4d240 0x56536ab5d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06b17fa640 0 client.16260 ms_handle_reset on v2:172.21.15.119:6836/1948234265 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 --2- 172.21.15.77:0/1539229935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56536ab4c850 0x56536ab5d3c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 >> 172.21.15.77:0/1539229935 conn(0x56536ab47dc0 msgr2=0x56536ab46750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 shutdown_connections 2024-03-16T09:45:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 1 -- 172.21.15.77:0/1539229935 wait complete. 2024-03-16T09:45:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d2e32740 20 client.16260 trim_cache size 0 max 16384 2024-03-16T09:45:09.098 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f06b40a2800 msgr2=0x7f06b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f06b40a2800 0x7f06b40a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x56536ab4daa0 tx=0x7f06c0004730 comp rx=0 tx=0).stop 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c81018f0 msgr2=0x7f06c8196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c81018f0 0x7f06c8196ea0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f06b800eb00 tx=0x7f06b8009f20 comp rx=0 tx=0).stop 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 shutdown_connections 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f06b40a2800 0x7f06b40a4cc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c81022b0 0x7f06c819df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06c81018f0 0x7f06c8196ea0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 --2- 172.21.15.77:0/2368478406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06c8100700 0x7f06c8196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 >> 172.21.15.77:0/2368478406 conn(0x7f06c80fbc90 msgr2=0x7f06c8104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.087+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 shutdown_connections 2024-03-16T09:45:09.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.088+0000 7f06d0169640 1 -- 172.21.15.77:0/2368478406 wait complete. 2024-03-16T09:45:09.100 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:08 smithi077 ceph-mon[32171]: pgmap v443: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 221 op/s 2024-03-16T09:45:09.100 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:08 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3498640686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:09.101 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T09:45:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:08 smithi119 ceph-mon[35632]: pgmap v443: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 221 op/s 2024-03-16T09:45:09.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:08 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3498640686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.202+0000 7f7c1a97e640 1 -- 172.21.15.77:0/1870583712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 msgr2=0x7f7c14100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.202+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/1870583712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c14100b00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009900 tx=0x7f7c0402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 -- 172.21.15.77:0/1870583712 shutdown_connections 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/1870583712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c14102290 0x7f7c14108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/1870583712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c141018d0 0x7f7c14101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/1870583712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c14100b00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 -- 172.21.15.77:0/1870583712 >> 172.21.15.77:0/1870583712 conn(0x7f7c140fbc70 msgr2=0x7f7c140fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 -- 172.21.15.77:0/1870583712 shutdown_connections 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 -- 172.21.15.77:0/1870583712 wait complete. 2024-03-16T09:45:09.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 Processor -- start 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 -- start start 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c141969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c141018d0 0x7f7c14196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.203+0000 7f7c1a97e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c14102290 0x7f7c1419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c1a97e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1410c200 con 0x7f7c141006e0 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c1a97e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1410c080 con 0x7f7c141018d0 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c1a97e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1410c380 con 0x7f7c14102290 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c141969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c141969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48790/0 (socket says 172.21.15.77:48790) 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 -- 172.21.15.77:0/3723778770 learned_addr learned my addr 172.21.15.77:0/3723778770 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c137fe640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c141018d0 0x7f7c14196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c14102290 0x7f7c1419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 -- 172.21.15.77:0/3723778770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c14102290 msgr2=0x7f7c1419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c14102290 0x7f7c1419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 -- 172.21.15.77:0/3723778770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c141018d0 msgr2=0x7f7c14196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c141018d0 0x7f7c14196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 -- 172.21.15.77:0/3723778770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1419e600 con 0x7f7c141006e0 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c13fff640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c141969e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7c04000c00 tx=0x7f7c04005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c117fa640 1 -- 172.21.15.77:0/3723778770 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04004e00 con 0x7f7c141006e0 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c117fa640 1 -- 172.21.15.77:0/3723778770 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c04005450 con 0x7f7c141006e0 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c14068940 con 0x7f7c141006e0 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c117fa640 1 -- 172.21.15.77:0/3723778770 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04047480 con 0x7f7c141006e0 2024-03-16T09:45:09.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.204+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c14068e90 con 0x7f7c141006e0 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.206+0000 7f7c1d56d740 1 Processor -- start 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.206+0000 7f7c117fa640 1 -- 172.21.15.77:0/3723778770 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7c04003680 con 0x7f7c141006e0 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c1d56d740 1 -- start start 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cce850 0x559393ccec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cd43b0 0x559393cd6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559393c33490 con 0x559393cd43b0 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559393bdbc90 con 0x559393ccf240 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559393c2f530 con 0x559393cce850 2024-03-16T09:45:09.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cd43b0 0x559393cd6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cd43b0 0x559393cd6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48796/0 (socket says 172.21.15.77:48796) 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 -- 172.21.15.77:0/1030051791 learned_addr learned my addr 172.21.15.77:0/1030051791 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c13fff640 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cce850 0x559393ccec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 -- 172.21.15.77:0/1030051791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cce850 msgr2=0x559393ccec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cce850 0x559393ccec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 -- 172.21.15.77:0/1030051791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 msgr2=0x559393cd3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 -- 172.21.15.77:0/1030051791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559393c96500 con 0x559393cd43b0 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c117fa640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7c000a2800 0x7f7c000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7c000a2800 0x7f7c000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cd43b0 0x559393cd6840 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009ca0 tx=0x7f7bfc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb7268163a8dbe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.207+0000 7f7c117fa640 1 -- 172.21.15.77:0/3723778770 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7c040f2b40 con 0x7f7c141006e0 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c0a7fc640 1 -- 172.21.15.77:0/1030051791 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc01e070 con 0x559393cd43b0 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c0a7fc640 1 -- 172.21.15.77:0/1030051791 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bfc019ae0 con 0x559393cd43b0 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c0a7fc640 1 -- 172.21.15.77:0/1030051791 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00e950 con 0x559393cd43b0 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1030051791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cd43b0 msgr2=0x559393cd6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cd43b0 0x559393cd6840 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009ca0 tx=0x7f7bfc011810 comp rx=0 tx=0).stop 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7c000a2800 0x7f7c000a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x559393ccfaa0 tx=0x7f7c0c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1030051791 shutdown_connections 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cd43b0 0x559393cd6840 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1030051791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cce850 0x559393ccec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1030051791 >> 172.21.15.77:0/1030051791 conn(0x559393cc9dc0 msgr2=0x559393cca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1030051791 shutdown_connections 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.208+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1030051791 wait complete. 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 -- start start 2024-03-16T09:45:09.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 Processor -- start 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cd43b0 0x559393ce0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559393cd7360 con 0x559393cce850 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559393bef230 con 0x559393ccf240 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559393cd74e0 con 0x559393cd43b0 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c1d56d740 10 client.0 _ll_register_callbacks cb 0x559393c2b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48806/0 (socket says 172.21.15.77:48806) 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c13fff640 1 -- 172.21.15.77:0/1943291000 learned_addr learned my addr 172.21.15.77:0/1943291000 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cd43b0 0x559393ce0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c13fff640 1 -- 172.21.15.77:0/1943291000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cd43b0 msgr2=0x559393ce0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c13fff640 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cd43b0 0x559393ce0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c13fff640 1 -- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 msgr2=0x559393cd92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.209+0000 7f7c13fff640 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c13fff640 1 -- 172.21.15.77:0/1943291000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559393bfe100 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c13fff640 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd8d90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7c0402c840 tx=0x7f7c04005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd5aceb83433c06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7bf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7bf37fe640 1 -- 172.21.15.77:0/1943291000 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040bb790 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7bf37fe640 1 -- 172.21.15.77:0/1943291000 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7c040c7a00 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bfc019ae0 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7bf37fe640 1 -- 172.21.15.77:0/1943291000 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7c040bee50 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 10 client.16272 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5593939473f0 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7bf37fe640 1 -- 172.21.15.77:0/1943291000 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bfc019ae0 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7bf37fe640 1 -- 172.21.15.77:0/1943291000 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7c0404f030 con 0x559393cce850 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 10 client.16272 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 10 client.16272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:45:09.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 20 client.16272 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393ce4b10 0x559393ce6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 4 client.16272 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559393c409d0 con 0x559393ce4b10 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.210+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393ce4b10 0x559393ce6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.211+0000 7f7bf37fe640 10 client.16272 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.211+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393ce4b10 0x559393ce6fa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c07ad7667fd1c57 server_cookie=6abcc3e572d0c692 in_seq=0 out_seq=0 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.211+0000 7f7bf37fe640 1 -- 172.21.15.77:0/1943291000 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559393c409d0 con 0x559393ce4b10 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.211+0000 7f7bf37fe640 10 client.16272 handle_command_reply: tid=0 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.211+0000 7f7c1d56d740 1 client.16272 shutdown 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 msgr2=0x559393cd8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd8d90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7c0402c840 tx=0x7f7c04005b70 comp rx=0 tx=0).stop 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 shutdown_connections 2024-03-16T09:45:09.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393ce4b10 0x559393ce6fa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393cd43b0 0x559393ce0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393ccf240 0x559393cd92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7bf37fe640 0 client.16272 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1943291000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd8d90 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 >> 172.21.15.77:0/1943291000 conn(0x559393cc9dc0 msgr2=0x559393cc8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 shutdown_connections 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1943291000 wait complete. 2024-03-16T09:45:09.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.212+0000 7f7c1d56d740 20 client.16272 trim_cache size 0 max 16384 2024-03-16T09:45:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 Processor -- start 2024-03-16T09:45:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 -- start start 2024-03-16T09:45:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393cc8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393cc8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141a9260 con 0x559393cce850 2024-03-16T09:45:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141a6b30 con 0x559393cd43b0 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141a4980 con 0x559393ccf240 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48808/0 (socket says 172.21.15.77:48808) 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 -- 172.21.15.77:0/1278254863 learned_addr learned my addr 172.21.15.77:0/1278254863 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393cc8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393cc8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 -- 172.21.15.77:0/1278254863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 msgr2=0x559393cc8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393cc8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 -- 172.21.15.77:0/1278254863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 msgr2=0x559393cc8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393cc8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c13fff640 1 -- 172.21.15.77:0/1278254863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559393a117d0 con 0x559393cce850 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393cc8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.225+0000 7f7c137fe640 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393cc8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:45:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c13fff640 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd90b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7c040bf480 tx=0x7f7c0402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596fffa62a337bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c09ffb640 1 -- 172.21.15.77:0/1278254863 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040bf030 con 0x559393cce850 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c09ffb640 1 -- 172.21.15.77:0/1278254863 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c040bb040 con 0x559393cce850 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c09ffb640 1 -- 172.21.15.77:0/1278254863 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040bf2c0 con 0x559393cce850 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1278254863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 msgr2=0x559393cd90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd90b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7c040bf480 tx=0x7f7c0402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1278254863 shutdown_connections 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393cc8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393cc8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/1278254863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393cd90b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1278254863 >> 172.21.15.77:0/1278254863 conn(0x559393cc9dc0 msgr2=0x559393cd89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1278254863 shutdown_connections 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.226+0000 7f7c1d56d740 1 -- 172.21.15.77:0/1278254863 wait complete. 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 -- start start 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 Processor -- start 2024-03-16T09:45:09.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393d45f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393d46470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393d4d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141a4ff0 con 0x559393cce850 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141a6460 con 0x559393cd43b0 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141a56c0 con 0x559393ccf240 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c1d56d740 10 client.0 _ll_register_callbacks cb 0x7f7c040057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393d45f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393d45f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48810/0 (socket says 172.21.15.77:48810) 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 -- 172.21.15.77:0/200300990 learned_addr learned my addr 172.21.15.77:0/200300990 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c137fe640 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393d46470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c18f1b640 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393d4d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 -- 172.21.15.77:0/200300990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 msgr2=0x559393d46470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393d46470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 -- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 msgr2=0x559393d4d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393d4d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.227+0000 7f7c13fff640 1 -- 172.21.15.77:0/200300990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559393d4daa0 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c13fff640 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393d45f30 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7c04049040 tx=0x7f7c040bda00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebd82a99ca793fc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7bf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7bf37fe640 1 -- 172.21.15.77:0/200300990 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040bb790 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7bf37fe640 1 -- 172.21.15.77:0/200300990 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7c040051a0 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559393c3ad20 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7bf37fe640 1 -- 172.21.15.77:0/200300990 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7c040a6050 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 10 client.16278 fetch_fsmap learned FSMap version 21 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559393d4daa0 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7bf37fe640 1 -- 172.21.15.77:0/200300990 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559393c3ad20 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7bf37fe640 1 -- 172.21.15.77:0/200300990 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7c0404f070 con 0x559393cce850 2024-03-16T09:45:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 10 client.16278 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 10 client.16278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 20 client.16278 populate_metadata read hostname 'smithi077' 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393d4f350 0x559393d51770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 4 client.16278 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 --> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559393bb86a0 con 0x559393d4f350 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.228+0000 7f7c137fe640 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393d4f350 0x559393d51770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7bf37fe640 10 client.16278 ms_handle_connect on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c137fe640 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393d4f350 0x559393d51770 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85a55ce04055cd61 server_cookie=6920ac961bc45970 in_seq=0 out_seq=0 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7bf37fe640 1 -- 172.21.15.77:0/200300990 <== mds.0 v2:172.21.15.119:6838/3029827381 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559393bb86a0 con 0x559393d4f350 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7bf37fe640 10 client.16278 handle_command_reply: tid=0 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 client.16278 shutdown 2024-03-16T09:45:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 msgr2=0x559393d45f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393d45f30 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7c04049040 tx=0x7f7c040bda00 comp rx=0 tx=0).stop 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 shutdown_connections 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:6838/3029827381,v1:172.21.15.119:6839/3029827381] conn(0x559393d4f350 0x559393d51770 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559393cd43b0 0x559393d4d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7bf37fe640 0 client.16278 ms_handle_reset on v2:172.21.15.119:6838/3029827381 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559393ccf240 0x559393d46470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 --2- 172.21.15.77:0/200300990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559393cce850 0x559393d45f30 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 >> 172.21.15.77:0/200300990 conn(0x559393cc9dc0 msgr2=0x559393ccd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 shutdown_connections 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 1 -- 172.21.15.77:0/200300990 wait complete. 2024-03-16T09:45:09.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.229+0000 7f7c1d56d740 20 client.16278 trim_cache size 0 max 16384 2024-03-16T09:45:09.241 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7c000a2800 msgr2=0x7f7c000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7c000a2800 0x7f7c000a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x559393ccfaa0 tx=0x7f7c0c009500 comp rx=0 tx=0).stop 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 msgr2=0x7f7c141969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c141969e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7c04000c00 tx=0x7f7c04005dc0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 shutdown_connections 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f7c000a2800 0x7f7c000a4cc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c14102290 0x7f7c1419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c141018d0 0x7f7c14196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 --2- 172.21.15.77:0/3723778770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c141006e0 0x7f7c141969e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 >> 172.21.15.77:0/3723778770 conn(0x7f7c140fbc70 msgr2=0x7f7c14104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 shutdown_connections 2024-03-16T09:45:09.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:09.230+0000 7f7c1a97e640 1 -- 172.21.15.77:0/3723778770 wait complete. 2024-03-16T09:45:09.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:08 smithi129 ceph-mon[35841]: pgmap v443: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 221 op/s 2024-03-16T09:45:09.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:08 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3498640686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:10.244 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-16T09:45:10.244 INFO:tasks.fwd_scrub:done joining 2024-03-16T09:45:10.245 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-16T09:45:10.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:45:10.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 -- 172.21.15.77:0/1743792194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4100780 msgr2=0x7f9dd4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 --2- 172.21.15.77:0/1743792194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4100780 0x7f9dd4100ba0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8009900 tx=0x7f9dc802bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 -- 172.21.15.77:0/1743792194 shutdown_connections 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 --2- 172.21.15.77:0/1743792194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dd4102330 0x7f9dd4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 --2- 172.21.15.77:0/1743792194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd4101970 0x7f9dd4101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 --2- 172.21.15.77:0/1743792194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4100780 0x7f9dd4100ba0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 -- 172.21.15.77:0/1743792194 >> 172.21.15.77:0/1743792194 conn(0x7f9dd40fbd10 msgr2=0x7f9dd40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 -- 172.21.15.77:0/1743792194 shutdown_connections 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.367+0000 7f9ddab77640 1 -- 172.21.15.77:0/1743792194 wait complete. 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 Processor -- start 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 -- start start 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd4100780 0x7f9dd4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4101970 0x7f9dd4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dd4102330 0x7f9dd419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd410d360 con 0x7f9dd4101970 2024-03-16T09:45:10.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd410d1e0 con 0x7f9dd4100780 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9ddab77640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd410d4e0 con 0x7f9dd4102330 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9dd3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4101970 0x7f9dd4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9dd3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4101970 0x7f9dd4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48824/0 (socket says 172.21.15.77:48824) 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9dd3fff640 1 -- 172.21.15.77:0/2169128958 learned_addr learned my addr 172.21.15.77:0/2169128958 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9dd8913640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd4100780 0x7f9dd4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.368+0000 7f9dd9114640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dd4102330 0x7f9dd419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd3fff640 1 -- 172.21.15.77:0/2169128958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dd4102330 msgr2=0x7f9dd419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd3fff640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dd4102330 0x7f9dd419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd3fff640 1 -- 172.21.15.77:0/2169128958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd4100780 msgr2=0x7f9dd4197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd3fff640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd4100780 0x7f9dd4197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd3fff640 1 -- 172.21.15.77:0/2169128958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd419f680 con 0x7f9dd4101970 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd3fff640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4101970 0x7f9dd4197fa0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000e9f0 tx=0x7f9dc000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd1ffb640 1 -- 172.21.15.77:0/2169128958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000cd80 con 0x7f9dd4101970 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd1ffb640 1 -- 172.21.15.77:0/2169128958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dc00108e0 con 0x7f9dd4101970 2024-03-16T09:45:10.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9dd1ffb640 1 -- 172.21.15.77:0/2169128958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0018920 con 0x7f9dd4101970 2024-03-16T09:45:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd419f970 con 0x7f9dd4101970 2024-03-16T09:45:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.369+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd419ff10 con 0x7f9dd4101970 2024-03-16T09:45:10.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.370+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd4107b40 con 0x7f9dd4101970 2024-03-16T09:45:10.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.371+0000 7f9dd1ffb640 1 -- 172.21.15.77:0/2169128958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9dc0010a90 con 0x7f9dd4101970 2024-03-16T09:45:10.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.372+0000 7f9dd1ffb640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9dc40a2880 0x7f9dc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.372+0000 7f9dd8913640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9dc40a2880 0x7f9dc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.372+0000 7f9dd1ffb640 1 -- 172.21.15.77:0/2169128958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9dc00cf130 con 0x7f9dd4101970 2024-03-16T09:45:10.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.372+0000 7f9dd8913640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9dc40a2880 0x7f9dc40a4d40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005ab0 tx=0x7f9dc8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:10.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.376+0000 7f9dd1ffb640 1 -- 172.21.15.77:0/2169128958 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9dc009bb90 con 0x7f9dd4101970 2024-03-16T09:45:10.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.553+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9dd402d400 con 0x7f9dd4101970 2024-03-16T09:45:10.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.554+0000 7f9dd1ffb640 1 -- 172.21.15.77:0/2169128958 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f9dc009fc30 con 0x7f9dd4101970 2024-03-16T09:45:10.558 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:10.558 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:45:10.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.556+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9dc40a2880 msgr2=0x7f9dc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.556+0000 7f9ddab77640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9dc40a2880 0x7f9dc40a4d40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005ab0 tx=0x7f9dc8005fd0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4101970 msgr2=0x7f9dd4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4101970 0x7f9dd4197fa0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000e9f0 tx=0x7f9dc000eeb0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 shutdown_connections 2024-03-16T09:45:10.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f9dc40a2880 0x7f9dc40a4d40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dd4102330 0x7f9dd419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dd4101970 0x7f9dd4197fa0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 --2- 172.21.15.77:0/2169128958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd4100780 0x7f9dd4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 >> 172.21.15.77:0/2169128958 conn(0x7f9dd40fbd10 msgr2=0x7f9dd4104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:10.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 shutdown_connections 2024-03-16T09:45:10.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.557+0000 7f9ddab77640 1 -- 172.21.15.77:0/2169128958 wait complete. 2024-03-16T09:45:10.571 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:45:10.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:45:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 -- 172.21.15.77:0/1112173918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4102b60 msgr2=0x7f61f4102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1112173918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4102b60 0x7f61f4102fe0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f61e4009930 tx=0x7f61e402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 -- 172.21.15.77:0/1112173918 shutdown_connections 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1112173918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61f4103520 0x7f61f4109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1112173918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4102b60 0x7f61f4102fe0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1112173918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f40ffaf0 0x7f61f40fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 -- 172.21.15.77:0/1112173918 >> 172.21.15.77:0/1112173918 conn(0x7f61f40fb3e0 msgr2=0x7f61f40fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 -- 172.21.15.77:0/1112173918 shutdown_connections 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.674+0000 7f61fb63d640 1 -- 172.21.15.77:0/1112173918 wait complete. 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 Processor -- start 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 -- start start 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61f40ffaf0 0x7f61f4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4102b60 0x7f61f4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4103520 0x7f61f419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f61f410d330 con 0x7f61f4103520 2024-03-16T09:45:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f61f410d1b0 con 0x7f61f40ffaf0 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f61f410d4b0 con 0x7f61f4102b60 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4103520 0x7f61f419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4103520 0x7f61f419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48834/0 (socket says 172.21.15.77:48834) 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 -- 172.21.15.77:0/1908585406 learned_addr learned my addr 172.21.15.77:0/1908585406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f93d9640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61f40ffaf0 0x7f61f4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f8bd8640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4102b60 0x7f61f4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 -- 172.21.15.77:0/1908585406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4102b60 msgr2=0x7f61f4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4102b60 0x7f61f4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 -- 172.21.15.77:0/1908585406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61f40ffaf0 msgr2=0x7f61f4197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61f40ffaf0 0x7f61f4197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 -- 172.21.15.77:0/1908585406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f419f6a0 con 0x7f61f4103520 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61f9bda640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4103520 0x7f61f419f030 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00efd0 tx=0x7f61ec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61ea7fc640 1 -- 172.21.15.77:0/1908585406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec003b70 con 0x7f61f4103520 2024-03-16T09:45:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61ea7fc640 1 -- 172.21.15.77:0/1908585406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61ec003d10 con 0x7f61f4103520 2024-03-16T09:45:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.675+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f419f990 con 0x7f61f4103520 2024-03-16T09:45:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.676+0000 7f61ea7fc640 1 -- 172.21.15.77:0/1908585406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec010980 con 0x7f61f4103520 2024-03-16T09:45:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.676+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f419fe60 con 0x7f61f4103520 2024-03-16T09:45:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.676+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f4108db0 con 0x7f61f4103520 2024-03-16T09:45:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.678+0000 7f61ea7fc640 1 -- 172.21.15.77:0/1908585406 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f61ec004b40 con 0x7f61f4103520 2024-03-16T09:45:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.678+0000 7f61ea7fc640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f61e00a2800 0x7f61e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.678+0000 7f61f93d9640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f61e00a2800 0x7f61e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.678+0000 7f61ea7fc640 1 -- 172.21.15.77:0/1908585406 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f61ec0cf230 con 0x7f61f4103520 2024-03-16T09:45:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.679+0000 7f61f93d9640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f61e00a2800 0x7f61e00a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f61f4069b30 tx=0x7f61f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.682+0000 7f61ea7fc640 1 -- 172.21.15.77:0/1908585406 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61ec09c2e0 con 0x7f61f4103520 2024-03-16T09:45:10.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.857+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f61f402d1b0 con 0x7f61f4103520 2024-03-16T09:45:10.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.857+0000 7f61ea7fc640 1 -- 172.21.15.77:0/1908585406 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f61ec09bbe0 con 0x7f61f4103520 2024-03-16T09:45:10.861 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.859+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f61e00a2800 msgr2=0x7f61e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.859+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f61e00a2800 0x7f61e00a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f61f4069b30 tx=0x7f61f0005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.859+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4103520 msgr2=0x7f61f419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.859+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4103520 0x7f61f419f030 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00efd0 tx=0x7f61ec00c6a0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 shutdown_connections 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f61e00a2800 0x7f61e00a4cc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61f4103520 0x7f61f419f030 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4102b60 0x7f61f4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 --2- 172.21.15.77:0/1908585406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61f40ffaf0 0x7f61f4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 >> 172.21.15.77:0/1908585406 conn(0x7f61f40fb3e0 msgr2=0x7f61f4105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:10.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 shutdown_connections 2024-03-16T09:45:10.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.860+0000 7f61fb63d640 1 -- 172.21.15.77:0/1908585406 wait complete. 2024-03-16T09:45:10.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:45:10.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.978+0000 7ffa505bc640 1 -- 172.21.15.77:0/1928039799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa48102290 msgr2=0x7ffa48108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.978+0000 7ffa505bc640 1 --2- 172.21.15.77:0/1928039799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa48102290 0x7ffa48108b20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ffa40009900 tx=0x7ffa4002bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- 172.21.15.77:0/1928039799 shutdown_connections 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 --2- 172.21.15.77:0/1928039799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa48102290 0x7ffa48108b20 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 --2- 172.21.15.77:0/1928039799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa481018d0 0x7ffa48101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 --2- 172.21.15.77:0/1928039799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa481006e0 0x7ffa48100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- 172.21.15.77:0/1928039799 >> 172.21.15.77:0/1928039799 conn(0x7ffa480fbc70 msgr2=0x7ffa480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- 172.21.15.77:0/1928039799 shutdown_connections 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- 172.21.15.77:0/1928039799 wait complete. 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 Processor -- start 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- start start 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa481006e0 0x7ffa48196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa481018d0 0x7ffa48196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa48102290 0x7ffa4819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810c1f0 con 0x7ffa481018d0 2024-03-16T09:45:10.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810c070 con 0x7ffa48102290 2024-03-16T09:45:10.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa505bc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4810c370 con 0x7ffa481006e0 2024-03-16T09:45:10.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.979+0000 7ffa4db57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa481018d0 0x7ffa48196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4e358640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa481006e0 0x7ffa48196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa481018d0 0x7ffa48196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48844/0 (socket says 172.21.15.77:48844) 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4e358640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa481006e0 0x7ffa48196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.77:52966/0 (socket says 172.21.15.77:52966) 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 -- 172.21.15.77:0/2442420345 learned_addr learned my addr 172.21.15.77:0/2442420345 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4eb59640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa48102290 0x7ffa4819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 -- 172.21.15.77:0/2442420345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa481006e0 msgr2=0x7ffa48196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa481006e0 0x7ffa48196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 -- 172.21.15.77:0/2442420345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa48102290 msgr2=0x7ffa4819ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa48102290 0x7ffa4819ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 -- 172.21.15.77:0/2442420345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4819e540 con 0x7ffa481018d0 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa4db57640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa481018d0 0x7ffa48196e60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800e9f0 tx=0x7ffa3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:10.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa3f7fe640 1 -- 172.21.15.77:0/2442420345 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800cd80 con 0x7ffa481018d0 2024-03-16T09:45:10.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4819e830 con 0x7ffa481018d0 2024-03-16T09:45:10.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa3f7fe640 1 -- 172.21.15.77:0/2442420345 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa380108e0 con 0x7ffa481018d0 2024-03-16T09:45:10.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa3f7fe640 1 -- 172.21.15.77:0/2442420345 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38018920 con 0x7ffa481018d0 2024-03-16T09:45:10.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.980+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4819edd0 con 0x7ffa481018d0 2024-03-16T09:45:10.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.981+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa48107a20 con 0x7ffa481018d0 2024-03-16T09:45:10.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.982+0000 7ffa3f7fe640 1 -- 172.21.15.77:0/2442420345 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ffa38010a90 con 0x7ffa481018d0 2024-03-16T09:45:10.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.983+0000 7ffa3f7fe640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa340a2800 0x7ffa340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:10.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.983+0000 7ffa4e358640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa340a2800 0x7ffa340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:10.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.983+0000 7ffa3f7fe640 1 -- 172.21.15.77:0/2442420345 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ffa380ced40 con 0x7ffa481018d0 2024-03-16T09:45:10.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.983+0000 7ffa4e358640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa340a2800 0x7ffa340a4cc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffa48101730 tx=0x7ffa44009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:10.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:10.987+0000 7ffa3f7fe640 1 -- 172.21.15.77:0/2442420345 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa3809b7a0 con 0x7ffa481018d0 2024-03-16T09:45:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.161+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa480623e0 con 0x7ffa481018d0 2024-03-16T09:45:11.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.162+0000 7ffa3f7fe640 1 -- 172.21.15.77:0/2442420345 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7ffa3809f840 con 0x7ffa481018d0 2024-03-16T09:45:11.165 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:45:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa340a2800 msgr2=0x7ffa340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa340a2800 0x7ffa340a4cc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffa48101730 tx=0x7ffa44009500 comp rx=0 tx=0).stop 2024-03-16T09:45:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa481018d0 msgr2=0x7ffa48196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa481018d0 0x7ffa48196e60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800e9f0 tx=0x7ffa3800eeb0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 shutdown_connections 2024-03-16T09:45:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ffa340a2800 0x7ffa340a4cc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa48102290 0x7ffa4819ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa481018d0 0x7ffa48196e60 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 --2- 172.21.15.77:0/2442420345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa481006e0 0x7ffa48196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 >> 172.21.15.77:0/2442420345 conn(0x7ffa480fbc70 msgr2=0x7ffa48104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:11.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 shutdown_connections 2024-03-16T09:45:11.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.164+0000 7ffa505bc640 1 -- 172.21.15.77:0/2442420345 wait complete. 2024-03-16T09:45:11.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:10 smithi077 ceph-mon[32171]: pgmap v444: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 220 op/s 2024-03-16T09:45:11.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:10 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2169128958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:11.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:10 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1908585406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:11.178 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:45:11.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:10 smithi119 ceph-mon[35632]: pgmap v444: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 220 op/s 2024-03-16T09:45:11.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:10 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2169128958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:11.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:10 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1908585406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 -- 172.21.15.77:0/27725766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a481006e0 msgr2=0x7f4a48100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/27725766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a481006e0 0x7f4a48100b00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009900 tx=0x7f4a4402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 -- 172.21.15.77:0/27725766 shutdown_connections 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/27725766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a48102290 0x7f4a48108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/27725766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a481018d0 0x7f4a48101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/27725766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a481006e0 0x7f4a48100b00 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 -- 172.21.15.77:0/27725766 >> 172.21.15.77:0/27725766 conn(0x7f4a480fbc70 msgr2=0x7f4a480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 -- 172.21.15.77:0/27725766 shutdown_connections 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.280+0000 7f4a4fe81640 1 -- 172.21.15.77:0/27725766 wait complete. 2024-03-16T09:45:11.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 Processor -- start 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 -- start start 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a481006e0 0x7f4a48197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a481018d0 0x7f4a48197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48102290 0x7f4a4819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4810c1e0 con 0x7f4a48102290 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4810c060 con 0x7f4a481018d0 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4fe81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4810c360 con 0x7f4a481006e0 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4e41e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48102290 0x7f4a4819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4dc1d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a481006e0 0x7f4a48197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4e41e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48102290 0x7f4a4819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48860/0 (socket says 172.21.15.77:48860) 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4e41e640 1 -- 172.21.15.77:0/921880108 learned_addr learned my addr 172.21.15.77:0/921880108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.281+0000 7f4a4d41c640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a481018d0 0x7f4a48197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4e41e640 1 -- 172.21.15.77:0/921880108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a481006e0 msgr2=0x7f4a48197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4e41e640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a481006e0 0x7f4a48197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4e41e640 1 -- 172.21.15.77:0/921880108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a481018d0 msgr2=0x7f4a48197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4e41e640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a481018d0 0x7f4a48197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4e41e640 1 -- 172.21.15.77:0/921880108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4819f6b0 con 0x7f4a48102290 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4dc1d640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a481006e0 0x7f4a48197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4e41e640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48102290 0x7f4a4819f040 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000eb10 tx=0x7f4a4000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a3effd640 1 -- 172.21.15.77:0/921880108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40004b20 con 0x7f4a48102290 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a3effd640 1 -- 172.21.15.77:0/921880108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a40004cc0 con 0x7f4a48102290 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4819f9a0 con 0x7f4a48102290 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a3effd640 1 -- 172.21.15.77:0/921880108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40010690 con 0x7f4a48102290 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.282+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a4819fea0 con 0x7f4a48102290 2024-03-16T09:45:11.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.283+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a481a0130 con 0x7f4a48102290 2024-03-16T09:45:11.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.284+0000 7f4a3effd640 1 -- 172.21.15.77:0/921880108 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4a40005020 con 0x7f4a48102290 2024-03-16T09:45:11.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.284+0000 7f4a3effd640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4a340a2800 0x7f4a340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.285+0000 7f4a4dc1d640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4a340a2800 0x7f4a340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.285+0000 7f4a3effd640 1 -- 172.21.15.77:0/921880108 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f4a400cefe0 con 0x7f4a48102290 2024-03-16T09:45:11.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.285+0000 7f4a4dc1d640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4a340a2800 0x7f4a340a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005ab0 tx=0x7f4a44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:11.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.289+0000 7f4a3effd640 1 -- 172.21.15.77:0/921880108 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a4009c090 con 0x7f4a48102290 2024-03-16T09:45:11.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:10 smithi129 ceph-mon[35841]: pgmap v444: 97 pgs: 97 active+clean; 974 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 220 op/s 2024-03-16T09:45:11.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:10 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2169128958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:11.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:10 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1908585406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.464+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4a481a0300 con 0x7f4a48102290 2024-03-16T09:45:11.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.464+0000 7f4a3effd640 1 -- 172.21.15.77:0/921880108 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f4a4009b990 con 0x7f4a48102290 2024-03-16T09:45:11.467 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:45:11.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4a340a2800 msgr2=0x7f4a340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4a340a2800 0x7f4a340a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005ab0 tx=0x7f4a44005fd0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48102290 msgr2=0x7f4a4819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48102290 0x7f4a4819f040 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000eb10 tx=0x7f4a4000c2d0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 shutdown_connections 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f4a340a2800 0x7f4a340a4cc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a48102290 0x7f4a4819f040 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a481018d0 0x7f4a48197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 --2- 172.21.15.77:0/921880108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a481006e0 0x7f4a48197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 >> 172.21.15.77:0/921880108 conn(0x7f4a480fbc70 msgr2=0x7f4a48069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 shutdown_connections 2024-03-16T09:45:11.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.466+0000 7f4a4fe81640 1 -- 172.21.15.77:0/921880108 wait complete. 2024-03-16T09:45:11.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-16T09:45:11.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 -- 172.21.15.77:0/1184081137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1018f0 msgr2=0x7f347c101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 --2- 172.21.15.77:0/1184081137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1018f0 0x7f347c101d70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3464009900 tx=0x7f346402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 -- 172.21.15.77:0/1184081137 shutdown_connections 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 --2- 172.21.15.77:0/1184081137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c1022b0 0x7f347c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 --2- 172.21.15.77:0/1184081137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1018f0 0x7f347c101d70 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 --2- 172.21.15.77:0/1184081137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f347c100700 0x7f347c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 -- 172.21.15.77:0/1184081137 >> 172.21.15.77:0/1184081137 conn(0x7f347c0fbc90 msgr2=0x7f347c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 -- 172.21.15.77:0/1184081137 shutdown_connections 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.579+0000 7f34817c6640 1 -- 172.21.15.77:0/1184081137 wait complete. 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 Processor -- start 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 -- start start 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f347c100700 0x7f347c06a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c1018f0 0x7f347c19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1022b0 0x7f347c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f347c10c230 con 0x7f347c1022b0 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f347c10c0b0 con 0x7f347c100700 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f34817c6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f347c10c3b0 con 0x7f347c1018f0 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1022b0 0x7f347c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347affd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f347c100700 0x7f347c06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1022b0 0x7f347c19ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48882/0 (socket says 172.21.15.77:48882) 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 -- 172.21.15.77:0/3257164987 learned_addr learned my addr 172.21.15.77:0/3257164987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f3473fff640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c1018f0 0x7f347c19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 -- 172.21.15.77:0/3257164987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c1018f0 msgr2=0x7f347c19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c1018f0 0x7f347c19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 -- 172.21.15.77:0/3257164987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f347c100700 msgr2=0x7f347c06a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f347c100700 0x7f347c06a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 -- 172.21.15.77:0/3257164987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c19f630 con 0x7f347c1022b0 2024-03-16T09:45:11.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.580+0000 7f347b7fe640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1022b0 0x7f347c19ed70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f347400eb10 tx=0x7f347400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:11.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.581+0000 7f3478ff9640 1 -- 172.21.15.77:0/3257164987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474004b20 con 0x7f347c1022b0 2024-03-16T09:45:11.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.581+0000 7f3478ff9640 1 -- 172.21.15.77:0/3257164987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3474004cc0 con 0x7f347c1022b0 2024-03-16T09:45:11.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.581+0000 7f3478ff9640 1 -- 172.21.15.77:0/3257164987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474010690 con 0x7f347c1022b0 2024-03-16T09:45:11.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.581+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347c19f920 con 0x7f347c1022b0 2024-03-16T09:45:11.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.581+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347c19fe40 con 0x7f347c1022b0 2024-03-16T09:45:11.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.582+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f347c107aa0 con 0x7f347c1022b0 2024-03-16T09:45:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.583+0000 7f3478ff9640 1 -- 172.21.15.77:0/3257164987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3474005020 con 0x7f347c1022b0 2024-03-16T09:45:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.583+0000 7f3478ff9640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34680a2800 0x7f34680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.583+0000 7f347affd640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34680a2800 0x7f34680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.583+0000 7f3478ff9640 1 -- 172.21.15.77:0/3257164987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f34740cf110 con 0x7f347c1022b0 2024-03-16T09:45:11.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.584+0000 7f347affd640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34680a2800 0x7f34680a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f347c101750 tx=0x7f346c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:11.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.587+0000 7f3478ff9640 1 -- 172.21.15.77:0/3257164987 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f347409c1c0 con 0x7f347c1022b0 2024-03-16T09:45:11.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.807+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f347c02d250 con 0x7f347c1022b0 2024-03-16T09:45:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.815+0000 7f3478ff9640 1 -- 172.21.15.77:0/3257164987 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+739830 (secure 0 0 0) 0x7f347409bac0 con 0x7f347c1022b0 2024-03-16T09:45:11.829 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:11.829 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:45:11.829 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 18, 2024-03-16T09:45:11.829 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.829 INFO:teuthology.orchestra.run.smithi077.stdout: "active_gid": 14152, 2024-03-16T09:45:11.829 INFO:teuthology.orchestra.run.smithi077.stdout: "active_name": "x", 2024-03-16T09:45:11.829 INFO:teuthology.orchestra.run.smithi077.stdout: "active_addrs": { 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:6800", 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3114858521 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:6801", 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3114858521 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.830 INFO:teuthology.orchestra.run.smithi077.stdout: "active_addr": "172.21.15.77:6801/3114858521", 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "active_change": "2024-03-16T09:32:57.693181+0000", 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "active_mgr_features": 4540701547738038271, 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "available": true, 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "standbys": [ 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 24109, 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "z", 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr_features": 4540701547738038271, 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "available_modules": [ 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "alerts", 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:11.831 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.833 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.834 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.835 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:11.836 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_destination": { 2024-03-16T09:45:11.837 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_destination", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email address to send alerts to", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_from_name": { 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_from_name", 2024-03-16T09:45:11.838 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "Ceph", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email From: name", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_host": { 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_host", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP server", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_password": { 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_password", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.840 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Password to authenticate with", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_port": { 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_port", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.841 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "465", 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP port", 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_sender": { 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_sender", 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.842 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP envelope sender", 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_ssl": { 2024-03-16T09:45:11.843 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_ssl", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_user": { 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_user", 2024-03-16T09:45:11.844 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "User to authenticate as", 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.845 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "balancer", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "active": { 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "active", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.846 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_time": { 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_time", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0000", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.847 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_weekday": { 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_weekday", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T09:45:11.848 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_max_iterations": { 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_max_iterations", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "25", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "250", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.849 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_metrics": { 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_metrics", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.850 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_step": { 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_step", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.5", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0.001", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "0.999", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "aggressiveness of optimization", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.851 INFO:teuthology.orchestra.run.smithi077.stdout: "end_time": { 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_time", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2359", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "end_weekday": { 2024-03-16T09:45:11.852 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_weekday", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.853 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.854 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.855 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.856 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.857 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "min_score": { 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_score", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.858 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.859 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "upmap", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "crush-compat", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "none", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "read", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap-read" 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Balancer mode", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_ids": { 2024-03-16T09:45:11.860 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_ids", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:11.861 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_deviation": { 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_deviation", 2024-03-16T09:45:11.862 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_optimizations": { 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_optimizations", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:11.863 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm", 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:11.864 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_down_multiplier": { 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_down_multiplier", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.865 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_refresh_rate": { 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_refresh_rate", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "20", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.866 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_starting_port": { 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_starting_port", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4721", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.867 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_ptrace": { 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_ptrace", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_interval": { 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_interval", 2024-03-16T09:45:11.868 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_memory_target_ratio": { 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.7", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm_log_destination": { 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm_log_destination", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "file", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "file,syslog", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "syslog" 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.871 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "cgroups_split": { 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cgroups_split", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.872 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "config_checks_enabled": { 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_checks_enabled", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.873 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "config_dashboard": { 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_dashboard", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.874 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_alertmanager": { 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_alertmanager", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_base": { 2024-03-16T09:45:11.875 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_base", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Container image name, without the tag", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_elasticsearch": { 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_elasticsearch", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.876 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "elasticsearch container image", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_grafana": { 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_grafana", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.877 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_haproxy": { 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_haproxy", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.878 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "HAproxy container image", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_agent": { 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_agent", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T09:45:11.879 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger agent container image", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_collector": { 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_collector", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.880 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger collector container image", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_query": { 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_query", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.881 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger query container image", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_keepalived": { 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_keepalived", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Keepalived container image", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_loki": { 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_loki", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Loki container image", 2024-03-16T09:45:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_node_exporter": { 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_node_exporter", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.884 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_nvmeof": { 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_nvmeof", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Nvme-of container image", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.885 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_prometheus": { 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_prometheus", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.886 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_promtail": { 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_promtail", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Promtail container image", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_snmp_gateway": { 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_snmp_gateway", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SNMP Gateway container image", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "container_init": { 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_init", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "daemon_cache_timeout": { 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "daemon_cache_timeout", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.890 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "default_cephadm_command_timeout": { 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "900", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.891 INFO:teuthology.orchestra.run.smithi077.stdout: "default_registry": { 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_registry", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "device_cache_timeout": { 2024-03-16T09:45:11.892 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_cache_timeout", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1800", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache device inventory", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "device_enhanced_scan": { 2024-03-16T09:45:11.893 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_enhanced_scan", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "facts_cache_timeout": { 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "facts_cache_timeout", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache host facts data", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.895 INFO:teuthology.orchestra.run.smithi077.stdout: "host_check_interval": { 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "host_check_interval", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to perform a host check", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "hw_monitoring": { 2024-03-16T09:45:11.896 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hw_monitoring", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "inventory_list_all": { 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "inventory_list_all", 2024-03-16T09:45:11.897 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:11.898 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.899 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "log_refresh_metadata": { 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_refresh_metadata", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.900 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.901 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.902 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.903 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.904 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "*", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T09:45:11.905 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "max_count_per_host": { 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_count_per_host", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of daemons per service per host", 2024-03-16T09:45:11.906 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "max_osd_draining_count": { 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_osd_draining_count", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.907 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "migration_current": { 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "migration_current", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "internal - do not modify", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.908 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "root", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm-package", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "root" 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.909 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "oob_default_addr": { 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "oob_default_addr", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "169.254.1.1", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.910 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "prometheus_alerts_path": { 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus_alerts_path", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.911 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_insecure": { 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_insecure", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.912 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_password": { 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_password", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_url": { 2024-03-16T09:45:11.913 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_url", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.914 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_username": { 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_username", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.915 INFO:teuthology.orchestra.run.smithi077.stdout: "secure_monitoring_stack": { 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "secure_monitoring_stack", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "service_discovery_port": { 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "service_discovery_port", 2024-03-16T09:45:11.916 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8765", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "cephadm service discovery port", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "ssh_config_file": { 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssh_config_file", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.917 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "use_agent": { 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_agent", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.918 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "use_repo_digest": { 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_repo_digest", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.919 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_failed_host_check": { 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_failed_host_check", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.920 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_daemons": { 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_daemons", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.921 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_hosts": { 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_hosts", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T09:45:11.922 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crash", 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.923 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.924 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.925 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.926 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.927 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "retain_interval": { 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retain_interval", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.928 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "31536000", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_recent_interval": { 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_recent_interval", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1209600", 2024-03-16T09:45:11.929 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dashboard", 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.931 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T09:45:11.932 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_ENABLED": { 2024-03-16T09:45:11.933 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T09:45:11.934 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T09:45:11.935 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.936 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.937 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.938 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.939 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.940 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.941 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.942 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.944 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.945 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_URL": { 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_URL", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.946 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.947 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.948 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.949 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.950 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "{}", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.952 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.953 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.954 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.957 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.958 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.959 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.960 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.961 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.962 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T09:45:11.963 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T09:45:11.964 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "45", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T09:45:11.965 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T09:45:11.966 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T09:45:11.967 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.968 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.969 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.971 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.972 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "cross_origin_url": { 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cross_origin_url", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.973 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "crt_file": { 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crt_file", 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.974 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "debug": { 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "debug", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.975 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable/disable debug options", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "jwt_token_ttl": { 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "jwt_token_ttl", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "28800", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.976 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.977 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.978 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:11.979 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.980 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.981 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.982 INFO:teuthology.orchestra.run.smithi077.stdout: "motd": { 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "motd", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "The message of the day", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect_resolve_ip_addr": { 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.983 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T09:45:11.984 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8080", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.986 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl_server_port": { 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl_server_port", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8443", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.987 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "redirect", 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.988 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect" 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.989 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "url_prefix": { 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url_prefix", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.990 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:11.991 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_monitoring": { 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_monitoring", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "monitor device health metrics", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.992 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.993 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.994 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.995 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:11.996 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "mark_out_threshold": { 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mark_out_threshold", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.997 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2419200", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_name": { 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_name", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.998 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "device_health_metrics", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "retention_period": { 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retention_period", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:11.999 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15552000", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain device health metrics", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_frequency": { 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_frequency", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T09:45:12.000 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "self_heal": { 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "self_heal", 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.001 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.002 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_threshold": { 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_threshold", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7257600", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.003 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "diskprediction_local", 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.004 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.005 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.006 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.007 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.008 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "predict_interval": { 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predict_interval", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T09:45:12.009 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "predictor_model": { 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predictor_model", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "prophetstor", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.010 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.011 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "influx", 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": false, 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "influxdb python module not found", 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "batch_size": { 2024-03-16T09:45:12.012 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "batch_size", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5000", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "database": { 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "database", 2024-03-16T09:45:12.013 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "ceph", 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": { 2024-03-16T09:45:12.014 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hostname", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server hostname", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.015 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "30", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "5", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.016 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.017 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.018 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.019 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.020 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "password": { 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "password", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.021 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "password of InfluxDB server user", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "port": { 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "port", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.022 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8086", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server port", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "false", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "threads": { 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threads", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.024 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "32", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "username": { 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "username", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.025 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "username of InfluxDB server user", 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "verify_ssl": { 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "verify_ssl", 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.026 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "true", 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "insights", 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.028 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.029 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.030 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.032 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "iostat", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.033 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.036 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.037 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "k8sevents", 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_event_retention_days": { 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ceph_event_retention_days", 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7", 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.038 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "config_check_secs": { 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_check_secs", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "10", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.039 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.040 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.041 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.042 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.043 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.044 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "localpool", 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "failure_domain": { 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "failure_domain", 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "host", 2024-03-16T09:45:12.045 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "failure domain for any created local pool", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.046 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.047 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.049 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.050 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "min_size": { 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_size", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default min_size for any created local pool", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T09:45:12.051 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "num_rep": { 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "num_rep", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default replica count for any created local pool", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.052 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_num": { 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_num", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "128", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.053 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "prefix": { 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prefix", 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name prefix for any created local pool", 2024-03-16T09:45:12.054 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "subtree": { 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "subtree", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "rack", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T09:45:12.055 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mds_autoscaler", 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.056 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.058 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.059 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.060 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.061 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirroring", 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.062 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.063 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.064 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.065 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.066 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.067 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "nfs", 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.068 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.069 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.070 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.071 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.072 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "fail_fs": { 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "fail_fs", 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.073 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.074 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.075 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.076 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.077 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.078 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator": { 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.079 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "rook", 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "test_orchestrator" 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Orchestrator backend", 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_perf_query", 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.080 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.081 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.082 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.084 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.085 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_support", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.086 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.087 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.088 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.089 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.090 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_autoscaler", 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.091 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.092 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.093 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.094 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.095 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.096 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "threshold": { 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threshold", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1.0", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "scaling threshold", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.097 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "progress", 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_pg_recovery_event": { 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_pg_recovery_event", 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.098 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.099 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.100 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.101 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.102 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.103 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.104 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "max_completed_events": { 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_completed_events", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "50", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "number of past completed events to remember", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.105 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long the module is going to sleep", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.106 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "cache": { 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cache", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.107 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "exclude_perf_counters": { 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "exclude_perf_counters", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.108 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.111 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.112 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.113 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools": { 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T09:45:12.114 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "300", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_interval": { 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_interval", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15.0", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.116 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.117 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "9283", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "stale_cache_strategy": { 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stale_cache_strategy", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.118 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "log", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.119 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "default", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "default", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "error" 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.120 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T09:45:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.122 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.123 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.124 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.125 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.126 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_snap_create": { 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_snap_create", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "mirror_snapshot_schedule": { 2024-03-16T09:45:12.127 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "trash_purge_schedule": { 2024-03-16T09:45:12.128 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "trash_purge_schedule", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.129 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "restful", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_auth": { 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_auth", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.130 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.131 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.132 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.134 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.135 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.136 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.137 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.138 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.139 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rgw", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.140 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.141 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.142 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.143 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.144 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rook", 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.145 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.146 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.147 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.148 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.149 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.150 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_class": { 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "storage_class", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "local", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.151 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "selftest", 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.152 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.153 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.154 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.155 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.156 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "roption1": { 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption1", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.157 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "roption2": { 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption2", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "xyz", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.158 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption1": { 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption1", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.159 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption2": { 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption2", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.160 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption3": { 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption3", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption4": { 2024-03-16T09:45:12.161 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption4", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption5": { 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption5", 2024-03-16T09:45:12.162 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption6": { 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption6", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.163 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption7": { 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption7", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.164 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "42", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "testkey": { 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testkey", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.165 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "testlkey": { 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testlkey", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.166 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "testnewline": { 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testnewline", 2024-03-16T09:45:12.167 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snap_schedule", 2024-03-16T09:45:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_m_granularity": { 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_m_granularity", 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.169 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "dump_on_update": { 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dump_on_update", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.170 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "dump database to debug log on update", 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.171 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.172 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.173 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.174 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.175 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.176 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stats", 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.177 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.178 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.179 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.180 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.181 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.182 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "status", 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.183 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.185 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telegraf", 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "address": { 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "address", 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.188 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.189 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.190 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.191 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.192 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.193 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.194 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telemetry", 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_basic": { 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_basic", 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.195 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_crash": { 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_crash", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.196 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_device": { 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_device", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.197 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_ident": { 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_ident", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.198 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_perf": { 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_perf", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.199 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "contact": { 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "contact", 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.200 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "description": { 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "description", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.201 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "device_url": { 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_url", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.202 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T09:45:12.203 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.204 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "24", 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "8", 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.205 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "last_opt_revision": { 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "last_opt_revision", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.206 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard": { 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.207 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard_description": { 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard_description", 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.208 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.210 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.210 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.210 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.210 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.210 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.210 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.210 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.211 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.211 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.211 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.211 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.211 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.211 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.211 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.212 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.213 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.214 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.215 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.215 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.215 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.215 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.215 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.215 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.215 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.216 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.217 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: "organization": { 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "organization", 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.218 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "proxy": { 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "proxy", 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "url": { 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url", 2024-03-16T09:45:12.220 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.221 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "test_orchestrator", 2024-03-16T09:45:12.222 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.222 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.222 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.222 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.222 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.222 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.223 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.224 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.225 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.226 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.227 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.228 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.229 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.230 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.231 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.232 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_clones": { 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_clones", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.233 INFO:teuthology.orchestra.run.smithi077.stdout: "periodic_async_work": { 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "periodic_async_work", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Periodically check for async work", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.234 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_delay": { 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_delay", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.235 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_no_wait": { 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.236 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix", 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "discovery_interval": { 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "discovery_interval", 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.237 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "100", 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "identifier": { 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "identifier", 2024-03-16T09:45:12.238 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.240 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.241 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.242 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.243 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.244 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_host": { 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_host", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.245 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_port": { 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_port", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10051", 2024-03-16T09:45:12.246 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_sender": { 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_sender", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.247 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 24110, 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "y", 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr_features": 4540701547738038271, 2024-03-16T09:45:12.248 INFO:teuthology.orchestra.run.smithi077.stdout: "available_modules": [ 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "alerts", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.249 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.250 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.251 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.252 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.253 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.254 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_destination": { 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_destination", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.255 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email address to send alerts to", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_from_name": { 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_from_name", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "Ceph", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email From: name", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_host": { 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_host", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.257 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP server", 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_password": { 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_password", 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.258 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Password to authenticate with", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_port": { 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_port", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "465", 2024-03-16T09:45:12.259 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP port", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_sender": { 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_sender", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.260 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP envelope sender", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_ssl": { 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_ssl", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.261 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_user": { 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_user", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.262 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "User to authenticate as", 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "balancer", 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "active": { 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "active", 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.263 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_time": { 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_time", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.264 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0000", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_weekday": { 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_weekday", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.265 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_max_iterations": { 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_max_iterations", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.266 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "25", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "250", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_metrics": { 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_metrics", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T09:45:12.267 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_step": { 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_step", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.5", 2024-03-16T09:45:12.268 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0.001", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "0.999", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "aggressiveness of optimization", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "end_time": { 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_time", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2359", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.269 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "end_weekday": { 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_weekday", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T09:45:12.270 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.273 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.274 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.275 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "min_score": { 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_score", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.276 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "upmap", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "crush-compat", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "none", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "read", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap", 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap-read" 2024-03-16T09:45:12.277 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Balancer mode", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_ids": { 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_ids", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.278 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T09:45:12.279 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_deviation": { 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_deviation", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.280 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_optimizations": { 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_optimizations", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.281 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_down_multiplier": { 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_down_multiplier", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.282 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_refresh_rate": { 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_refresh_rate", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "20", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T09:45:12.283 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_starting_port": { 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_starting_port", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4721", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.284 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_ptrace": { 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_ptrace", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T09:45:12.285 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_interval": { 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_interval", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.286 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_memory_target_ratio": { 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.7", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.287 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm_log_destination": { 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm_log_destination", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "file", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "file,syslog", 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "syslog" 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.288 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "cgroups_split": { 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cgroups_split", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T09:45:12.289 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "config_checks_enabled": { 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_checks_enabled", 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.290 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "config_dashboard": { 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_dashboard", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.291 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_alertmanager": { 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_alertmanager", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.292 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_base": { 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_base", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Container image name, without the tag", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.293 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_elasticsearch": { 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_elasticsearch", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "elasticsearch container image", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.294 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_grafana": { 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_grafana", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.295 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_haproxy": { 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_haproxy", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "HAproxy container image", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.296 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_agent": { 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_agent", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger agent container image", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.297 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_collector": { 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_collector", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger collector container image", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.298 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_query": { 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_query", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger query container image", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.299 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_keepalived": { 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_keepalived", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Keepalived container image", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.300 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_loki": { 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_loki", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Loki container image", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_node_exporter": { 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_node_exporter", 2024-03-16T09:45:12.301 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_nvmeof": { 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_nvmeof", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.302 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Nvme-of container image", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_prometheus": { 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_prometheus", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T09:45:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_promtail": { 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_promtail", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.304 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Promtail container image", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_snmp_gateway": { 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_snmp_gateway", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.305 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SNMP Gateway container image", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "container_init": { 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_init", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.306 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "daemon_cache_timeout": { 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "daemon_cache_timeout", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T09:45:12.307 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "default_cephadm_command_timeout": { 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "900", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T09:45:12.308 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "default_registry": { 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_registry", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.309 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "device_cache_timeout": { 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_cache_timeout", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1800", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache device inventory", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.310 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "device_enhanced_scan": { 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_enhanced_scan", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.311 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "facts_cache_timeout": { 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "facts_cache_timeout", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache host facts data", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.312 INFO:teuthology.orchestra.run.smithi077.stdout: "host_check_interval": { 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "host_check_interval", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to perform a host check", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.313 INFO:teuthology.orchestra.run.smithi077.stdout: "hw_monitoring": { 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hw_monitoring", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "inventory_list_all": { 2024-03-16T09:45:12.314 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "inventory_list_all", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.315 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.316 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "log_refresh_metadata": { 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_refresh_metadata", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.317 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.318 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.319 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.320 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.321 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.322 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "*", 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "max_count_per_host": { 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_count_per_host", 2024-03-16T09:45:12.323 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of daemons per service per host", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "max_osd_draining_count": { 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_osd_draining_count", 2024-03-16T09:45:12.324 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "migration_current": { 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "migration_current", 2024-03-16T09:45:12.325 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "internal - do not modify", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T09:45:12.326 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "root", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm-package", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "root" 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.327 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "oob_default_addr": { 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "oob_default_addr", 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "169.254.1.1", 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T09:45:12.328 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "prometheus_alerts_path": { 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus_alerts_path", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.329 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_insecure": { 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_insecure", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.330 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_password": { 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_password", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.331 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_url": { 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_url", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_username": { 2024-03-16T09:45:12.332 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_username", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "secure_monitoring_stack": { 2024-03-16T09:45:12.333 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "secure_monitoring_stack", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "service_discovery_port": { 2024-03-16T09:45:12.334 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "service_discovery_port", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8765", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "cephadm service discovery port", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "ssh_config_file": { 2024-03-16T09:45:12.335 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssh_config_file", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "use_agent": { 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_agent", 2024-03-16T09:45:12.336 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "use_repo_digest": { 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_repo_digest", 2024-03-16T09:45:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_failed_host_check": { 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_failed_host_check", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_daemons": { 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_daemons", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_hosts": { 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_hosts", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.340 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crash", 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.341 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.342 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.343 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.344 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.345 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "retain_interval": { 2024-03-16T09:45:12.346 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retain_interval", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "31536000", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_recent_interval": { 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_recent_interval", 2024-03-16T09:45:12.347 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1209600", 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.348 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dashboard", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.349 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.350 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.351 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_ENABLED": { 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.352 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.353 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.354 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T09:45:12.355 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T09:45:12.356 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T09:45:12.357 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T09:45:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.359 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.360 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.361 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.362 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.363 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_URL": { 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_URL", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.364 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.365 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.366 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.367 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.368 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.369 INFO:teuthology.orchestra.run.smithi077.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "{}", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.370 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.371 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.372 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.373 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.374 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.375 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.376 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T09:45:12.377 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T09:45:12.378 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T09:45:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.381 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "45", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.383 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.385 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.386 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "cross_origin_url": { 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cross_origin_url", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: "crt_file": { 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crt_file", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: "debug": { 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "debug", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable/disable debug options", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "jwt_token_ttl": { 2024-03-16T09:45:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "jwt_token_ttl", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "28800", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.402 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "motd": { 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "motd", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "The message of the day", 2024-03-16T09:45:12.403 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect_resolve_ip_addr": { 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.404 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.405 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8080", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.406 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl_server_port": { 2024-03-16T09:45:12.407 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl_server_port", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8443", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T09:45:12.408 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "redirect", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect" 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.409 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.410 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "url_prefix": { 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url_prefix", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.411 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_monitoring": { 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_monitoring", 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.412 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "monitor device health metrics", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.414 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.416 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.417 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "mark_out_threshold": { 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mark_out_threshold", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2419200", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T09:45:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_name": { 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_name", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "device_health_metrics", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.419 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "retention_period": { 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retention_period", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15552000", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain device health metrics", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.420 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_frequency": { 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_frequency", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.421 INFO:teuthology.orchestra.run.smithi077.stdout: "self_heal": { 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "self_heal", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.422 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.423 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_threshold": { 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_threshold", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7257600", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.424 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "diskprediction_local", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.425 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.426 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.427 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.428 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.429 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "predict_interval": { 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predict_interval", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.430 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "predictor_model": { 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predictor_model", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "prophetstor", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.431 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.432 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "influx", 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": false, 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "influxdb python module not found", 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "batch_size": { 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "batch_size", 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5000", 2024-03-16T09:45:12.433 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "database": { 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "database", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "ceph", 2024-03-16T09:45:12.434 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": { 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hostname", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.435 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server hostname", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "30", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "5", 2024-03-16T09:45:12.436 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.438 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.439 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.440 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.441 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "password": { 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "password", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.442 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "password of InfluxDB server user", 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "port": { 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "port", 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8086", 2024-03-16T09:45:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server port", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "false", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.444 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "threads": { 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threads", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "32", 2024-03-16T09:45:12.445 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "username": { 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "username", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.446 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "username of InfluxDB server user", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "verify_ssl": { 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "verify_ssl", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "true", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.447 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "insights", 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.448 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.449 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.450 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.451 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.452 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.453 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "iostat", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.454 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.455 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.457 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.458 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "k8sevents", 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_event_retention_days": { 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ceph_event_retention_days", 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.459 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "config_check_secs": { 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_check_secs", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.460 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "10", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.461 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.462 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.463 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.464 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "localpool", 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "failure_domain": { 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "failure_domain", 2024-03-16T09:45:12.466 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "host", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "failure domain for any created local pool", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.469 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.470 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.471 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "min_size": { 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_size", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.472 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default min_size for any created local pool", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "num_rep": { 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "num_rep", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.473 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default replica count for any created local pool", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_num": { 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_num", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "128", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.474 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "prefix": { 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prefix", 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.475 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name prefix for any created local pool", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "subtree": { 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "subtree", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.476 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "rack", 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mds_autoscaler", 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.477 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.478 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.479 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.481 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.482 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirroring", 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.483 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.484 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.485 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.486 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.487 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "nfs", 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.488 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.489 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.490 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.491 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.492 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "fail_fs": { 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "fail_fs", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T09:45:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.495 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.496 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.498 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.499 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator": { 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "rook", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "test_orchestrator" 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Orchestrator backend", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.500 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_perf_query", 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.501 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.503 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.504 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.505 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_support", 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.506 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.507 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.508 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.509 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.510 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_autoscaler", 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.511 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.516 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "threshold": { 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threshold", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1.0", 2024-03-16T09:45:12.517 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "scaling threshold", 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "progress", 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.518 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_pg_recovery_event": { 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_pg_recovery_event", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.520 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.521 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.522 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.523 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "max_completed_events": { 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_completed_events", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "50", 2024-03-16T09:45:12.525 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "number of past completed events to remember", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.526 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long the module is going to sleep", 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus", 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "cache": { 2024-03-16T09:45:12.527 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cache", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "exclude_perf_counters": { 2024-03-16T09:45:12.528 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "exclude_perf_counters", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.529 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.530 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.531 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.532 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.533 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.534 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools": { 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T09:45:12.535 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "300", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_interval": { 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_interval", 2024-03-16T09:45:12.536 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15.0", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.537 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.538 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "9283", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "stale_cache_strategy": { 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stale_cache_strategy", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.539 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "log", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "default", 2024-03-16T09:45:12.540 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "default", 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "error" 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.541 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.542 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.543 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.544 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.547 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_snap_create": { 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_snap_create", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.548 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "mirror_snapshot_schedule": { 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.549 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "trash_purge_schedule": { 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "trash_purge_schedule", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.550 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "restful", 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_auth": { 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_auth", 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.551 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.552 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.553 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.554 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.555 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.556 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.557 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.558 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.559 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rgw", 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.560 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.561 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.562 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.563 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.564 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rook", 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.565 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.567 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.568 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.569 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.570 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_class": { 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "storage_class", 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.572 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "local", 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.573 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "selftest", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.575 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.576 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.577 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.578 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "roption1": { 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption1", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.579 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "roption2": { 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption2", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "xyz", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.580 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption1": { 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption1", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.581 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption2": { 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption2", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.582 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption3": { 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption3", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.583 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption4": { 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption4", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption5": { 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption5", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.585 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption6": { 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption6", 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.586 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption7": { 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption7", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "42", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.587 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "testkey": { 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testkey", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.588 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "testlkey": { 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testlkey", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.589 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "testnewline": { 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testnewline", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snap_schedule", 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_m_granularity": { 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_m_granularity", 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.591 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "dump_on_update": { 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dump_on_update", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.592 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "dump database to debug log on update", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.593 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.594 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.595 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.596 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stats", 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.598 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.599 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.600 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.601 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.601 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.601 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.601 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.601 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.601 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.601 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.602 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.602 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.602 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.602 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.602 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.602 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.603 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.604 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "status", 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.605 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.606 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.607 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.608 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.609 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telegraf", 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.610 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "address": { 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "address", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.611 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.613 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.614 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.616 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.617 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telemetry", 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_basic": { 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_basic", 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.618 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_crash": { 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_crash", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.619 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_device": { 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_device", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.620 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_ident": { 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_ident", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.621 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_perf": { 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_perf", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.622 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "contact": { 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "contact", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.623 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "description": { 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "description", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.624 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "device_url": { 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_url", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.625 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "24", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "8", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "last_opt_revision": { 2024-03-16T09:45:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "last_opt_revision", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard": { 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard", 2024-03-16T09:45:12.628 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard_description": { 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard_description", 2024-03-16T09:45:12.629 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.630 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.631 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.632 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.633 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.634 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "organization": { 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "organization", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.635 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "proxy": { 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "proxy", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.636 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "url": { 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.637 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "test_orchestrator", 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.638 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.639 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.640 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.641 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.642 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.644 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.645 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.646 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.647 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_clones": { 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_clones", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.648 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "periodic_async_work": { 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "periodic_async_work", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.649 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Periodically check for async work", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_delay": { 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_delay", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_no_wait": { 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix", 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.652 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "discovery_interval": { 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "discovery_interval", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "100", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.653 INFO:teuthology.orchestra.run.smithi077.stdout: "identifier": { 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "identifier", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.654 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.655 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.656 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.657 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.658 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.659 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_host": { 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_host", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.660 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_port": { 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_port", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10051", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.661 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_sender": { 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_sender", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.662 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "modules": [ 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "dashboard", 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "iostat", 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "nfs", 2024-03-16T09:45:12.663 INFO:teuthology.orchestra.run.smithi077.stdout: "restful" 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "available_modules": [ 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "alerts", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.664 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.665 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.666 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.667 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.668 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_destination": { 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_destination", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email address to send alerts to", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.670 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_from_name": { 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_from_name", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "Ceph", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email From: name", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.671 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_host": { 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_host", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP server", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.672 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_password": { 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_password", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Password to authenticate with", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_port": { 2024-03-16T09:45:12.673 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_port", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "465", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP port", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_sender": { 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_sender", 2024-03-16T09:45:12.674 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP envelope sender", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_ssl": { 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_ssl", 2024-03-16T09:45:12.675 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_user": { 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_user", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.676 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "User to authenticate as", 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "balancer", 2024-03-16T09:45:12.677 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "active": { 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "active", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.678 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_time": { 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_time", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0000", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_weekday": { 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_weekday", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.680 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_max_iterations": { 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_max_iterations", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "25", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "250", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.681 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_metrics": { 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_metrics", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T09:45:12.682 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_step": { 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_step", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.5", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0.001", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "0.999", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "aggressiveness of optimization", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T09:45:12.683 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "end_time": { 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_time", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2359", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T09:45:12.684 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "end_weekday": { 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_weekday", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T09:45:12.685 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.686 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.687 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.689 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "min_score": { 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_score", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.690 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.691 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "upmap", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "crush-compat", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "none", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "read", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap-read" 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Balancer mode", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.692 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_ids": { 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_ids", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.694 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_deviation": { 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_deviation", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.695 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_optimizations": { 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_optimizations", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.696 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm", 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_down_multiplier": { 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_down_multiplier", 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T09:45:12.697 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_refresh_rate": { 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_refresh_rate", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "20", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.698 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_starting_port": { 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_starting_port", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4721", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.699 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_ptrace": { 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_ptrace", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.700 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_interval": { 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_interval", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.701 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_memory_target_ratio": { 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.7", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T09:45:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm_log_destination": { 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm_log_destination", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "file", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "file,syslog", 2024-03-16T09:45:12.703 INFO:teuthology.orchestra.run.smithi077.stdout: "syslog" 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "cgroups_split": { 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cgroups_split", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.704 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "config_checks_enabled": { 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_checks_enabled", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T09:45:12.705 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "config_dashboard": { 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_dashboard", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T09:45:12.706 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_alertmanager": { 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_alertmanager", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.707 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_base": { 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_base", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Container image name, without the tag", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.708 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_elasticsearch": { 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_elasticsearch", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "elasticsearch container image", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.709 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_grafana": { 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_grafana", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_haproxy": { 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_haproxy", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "HAproxy container image", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_agent": { 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_agent", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger agent container image", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.712 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_collector": { 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_collector", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger collector container image", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.713 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_query": { 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_query", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger query container image", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.714 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_keepalived": { 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_keepalived", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Keepalived container image", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.715 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_loki": { 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_loki", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Loki container image", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_node_exporter": { 2024-03-16T09:45:12.716 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_node_exporter", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_nvmeof": { 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_nvmeof", 2024-03-16T09:45:12.717 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Nvme-of container image", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_prometheus": { 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_prometheus", 2024-03-16T09:45:12.718 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_promtail": { 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_promtail", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.719 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Promtail container image", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_snmp_gateway": { 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_snmp_gateway", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T09:45:12.720 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SNMP Gateway container image", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "container_init": { 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_init", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.721 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "daemon_cache_timeout": { 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "daemon_cache_timeout", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.722 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "default_cephadm_command_timeout": { 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "900", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T09:45:12.723 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "default_registry": { 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_registry", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T09:45:12.724 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "device_cache_timeout": { 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_cache_timeout", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1800", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache device inventory", 2024-03-16T09:45:12.725 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "device_enhanced_scan": { 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_enhanced_scan", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T09:45:12.726 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "facts_cache_timeout": { 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "facts_cache_timeout", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache host facts data", 2024-03-16T09:45:12.727 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "host_check_interval": { 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "host_check_interval", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to perform a host check", 2024-03-16T09:45:12.728 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "hw_monitoring": { 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hw_monitoring", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T09:45:12.729 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "inventory_list_all": { 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "inventory_list_all", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.730 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.731 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "log_refresh_metadata": { 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_refresh_metadata", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.732 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.733 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.735 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T09:45:12.736 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T09:45:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "*", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "max_count_per_host": { 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_count_per_host", 2024-03-16T09:45:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of daemons per service per host", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "max_osd_draining_count": { 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_osd_draining_count", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.739 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "migration_current": { 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "migration_current", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.740 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "internal - do not modify", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.741 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "root", 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm-package", 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "root" 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "oob_default_addr": { 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "oob_default_addr", 2024-03-16T09:45:12.742 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "169.254.1.1", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "prometheus_alerts_path": { 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus_alerts_path", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.743 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_insecure": { 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_insecure", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.744 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_password": { 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_password", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.745 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_url": { 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_url", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.746 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_username": { 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_username", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.747 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "secure_monitoring_stack": { 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "secure_monitoring_stack", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.748 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "service_discovery_port": { 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "service_discovery_port", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8765", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "cephadm service discovery port", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.749 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "ssh_config_file": { 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssh_config_file", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "use_agent": { 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_agent", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.751 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "use_repo_digest": { 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_repo_digest", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.752 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_failed_host_check": { 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_failed_host_check", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.753 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_daemons": { 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_daemons", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_hosts": { 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_hosts", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.755 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crash", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.758 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.759 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.760 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "retain_interval": { 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retain_interval", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "31536000", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.761 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_recent_interval": { 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_recent_interval", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1209600", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.762 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dashboard", 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.763 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.764 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_ENABLED": { 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.766 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.767 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_URL": { 2024-03-16T09:45:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_URL", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "{}", 2024-03-16T09:45:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.787 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.789 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.790 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.791 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T09:45:12.792 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.793 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.794 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.795 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.796 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "45", 2024-03-16T09:45:12.797 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.798 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T09:45:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.800 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.801 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.802 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.803 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.804 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "cross_origin_url": { 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cross_origin_url", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.806 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "crt_file": { 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crt_file", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.807 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "debug": { 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "debug", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable/disable debug options", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "jwt_token_ttl": { 2024-03-16T09:45:12.808 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "jwt_token_ttl", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "28800", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T09:45:12.809 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.812 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.813 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.814 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "motd": { 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "motd", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.815 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "The message of the day", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect_resolve_ip_addr": { 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.816 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8080", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.818 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.819 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl_server_port": { 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl_server_port", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8443", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.820 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "redirect", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect" 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.821 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.822 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "url_prefix": { 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url_prefix", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.823 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_monitoring": { 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_monitoring", 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.824 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "monitor device health metrics", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.826 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.827 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.828 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.829 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "mark_out_threshold": { 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mark_out_threshold", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2419200", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.830 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_name": { 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_name", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "device_health_metrics", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T09:45:12.831 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "retention_period": { 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retention_period", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15552000", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain device health metrics", 2024-03-16T09:45:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_frequency": { 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_frequency", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T09:45:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "self_heal": { 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "self_heal", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.834 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.835 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_threshold": { 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_threshold", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7257600", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.836 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "diskprediction_local", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.837 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.838 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.839 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.840 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.841 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "predict_interval": { 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predict_interval", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.842 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "predictor_model": { 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predictor_model", 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "prophetstor", 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.843 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.844 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "influx", 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": false, 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "influxdb python module not found", 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "batch_size": { 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "batch_size", 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.845 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5000", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "database": { 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "database", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.846 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "ceph", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": { 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hostname", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.847 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server hostname", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.848 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "30", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "5", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.849 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.852 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.853 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "password": { 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "password", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "password of InfluxDB server user", 2024-03-16T09:45:12.854 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "port": { 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "port", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8086", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server port", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.855 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "false", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.856 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "threads": { 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threads", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "32", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.857 INFO:teuthology.orchestra.run.smithi077.stdout: "username": { 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "username", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "username of InfluxDB server user", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.858 INFO:teuthology.orchestra.run.smithi077.stdout: "verify_ssl": { 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "verify_ssl", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "true", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.859 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "insights", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.860 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.861 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.862 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.863 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.864 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "iostat", 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.865 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.866 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.867 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.868 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "k8sevents", 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_event_retention_days": { 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ceph_event_retention_days", 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.870 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "config_check_secs": { 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_check_secs", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.871 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "10", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.872 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.873 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.874 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.875 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "localpool", 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "failure_domain": { 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "failure_domain", 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.877 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "host", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "failure domain for any created local pool", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.878 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.879 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.880 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.881 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.882 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "min_size": { 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_size", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.883 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default min_size for any created local pool", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "num_rep": { 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "num_rep", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default replica count for any created local pool", 2024-03-16T09:45:12.884 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_num": { 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_num", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "128", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T09:45:12.885 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "prefix": { 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prefix", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name prefix for any created local pool", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.886 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "subtree": { 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "subtree", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "rack", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.887 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mds_autoscaler", 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.888 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.889 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.890 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.891 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.892 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirroring", 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.893 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.896 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.897 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "nfs", 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.898 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.899 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.900 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.901 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "fail_fs": { 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "fail_fs", 2024-03-16T09:45:12.903 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.905 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.907 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.908 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator": { 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T09:45:12.909 INFO:teuthology.orchestra.run.smithi077.stdout: "rook", 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "test_orchestrator" 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Orchestrator backend", 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_perf_query", 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.910 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.911 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.912 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.913 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.914 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_support", 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.915 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.916 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.917 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.918 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.919 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_autoscaler", 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.921 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.922 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.923 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.924 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.925 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "threshold": { 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threshold", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.926 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1.0", 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "scaling threshold", 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "progress", 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_pg_recovery_event": { 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_pg_recovery_event", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.928 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.929 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.930 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.931 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.932 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.933 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "max_completed_events": { 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_completed_events", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.934 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "50", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "number of past completed events to remember", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.935 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long the module is going to sleep", 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus", 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.936 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "cache": { 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cache", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.937 INFO:teuthology.orchestra.run.smithi077.stdout: "exclude_perf_counters": { 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "exclude_perf_counters", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.938 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.939 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.940 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.941 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.942 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.943 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools": { 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools", 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "300", 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.945 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_interval": { 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_interval", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15.0", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.946 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.947 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "9283", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.948 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "stale_cache_strategy": { 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stale_cache_strategy", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "log", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.949 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "default", 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "default", 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "error" 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.950 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.951 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.952 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.953 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.954 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.955 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_snap_create": { 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_snap_create", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.957 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "mirror_snapshot_schedule": { 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.958 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "trash_purge_schedule": { 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "trash_purge_schedule", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.959 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "restful", 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_auth": { 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_auth", 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.960 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.961 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.962 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.963 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.964 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.965 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.966 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.967 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.968 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rgw", 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.970 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.971 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.972 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.973 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rook", 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.974 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.975 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.976 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.977 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.978 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_class": { 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "storage_class", 2024-03-16T09:45:12.979 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "local", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "selftest", 2024-03-16T09:45:12.980 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.981 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.982 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.983 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:12.984 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "roption1": { 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption1", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.985 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "roption2": { 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption2", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.986 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "xyz", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption1": { 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption1", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.987 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption2": { 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption2", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.988 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption3": { 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption3", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.989 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption4": { 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption4", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.990 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption5": { 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption5", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.991 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption6": { 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption6", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.992 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption7": { 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption7", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "42", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.993 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "testkey": { 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testkey", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.994 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "testlkey": { 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testlkey", 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.995 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "testnewline": { 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testnewline", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.996 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snap_schedule", 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_m_granularity": { 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_m_granularity", 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.997 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "dump_on_update": { 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dump_on_update", 2024-03-16T09:45:12.998 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "dump database to debug log on update", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:12.999 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.000 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.001 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.002 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.003 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stats", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.005 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.006 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.007 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.008 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.009 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "status", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.010 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.011 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.013 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.014 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.015 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.016 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telegraf", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "address": { 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "address", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.017 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.018 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.019 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.020 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.021 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.022 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telemetry", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_basic": { 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_basic", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T09:45:13.024 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_crash": { 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_crash", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.025 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_device": { 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_device", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.026 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_ident": { 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_ident", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.027 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_perf": { 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_perf", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.028 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "contact": { 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "contact", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.029 INFO:teuthology.orchestra.run.smithi077.stdout: "description": { 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "description", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "device_url": { 2024-03-16T09:45:13.030 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_url", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.031 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.032 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "24", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "8", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "last_opt_revision": { 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "last_opt_revision", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.033 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard": { 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.034 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard_description": { 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard_description", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.035 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.036 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.037 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.038 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.039 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.040 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "organization": { 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "organization", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "proxy": { 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "proxy", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.042 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "url": { 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.043 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "test_orchestrator", 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.044 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.046 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.047 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.048 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.049 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.050 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.051 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.052 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.053 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_clones": { 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_clones", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T09:45:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "periodic_async_work": { 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "periodic_async_work", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Periodically check for async work", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_delay": { 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_delay", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.056 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_no_wait": { 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.057 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix", 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "discovery_interval": { 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "discovery_interval", 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "100", 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.058 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "identifier": { 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "identifier", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.059 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.060 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.061 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T09:45:13.062 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.063 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.064 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.065 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_host": { 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_host", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.066 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_port": { 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_port", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10051", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_sender": { 2024-03-16T09:45:13.067 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_sender", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.068 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "services": { 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "dashboard": "https://172.21.15.77:8443/" 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "always_on_modules": { 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "octopus": [ 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T09:45:13.069 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "pacific": [ 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T09:45:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "quincy": [ 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "reef": [ 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T09:45:13.071 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "squid": [ 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T09:45:13.072 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "last_failure_osd_epoch": 3, 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "active_clients": [ 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1858843655 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.073 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "libcephsqlite", 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 2486423961 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T09:45:13.074 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3932809728 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 510540831 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T09:45:13.075 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:45:13.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:11 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2442420345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:13.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:11 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/921880108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:13.077 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:11 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3257164987' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T09:45:13.077 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:11 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2442420345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:13.077 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:11 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/921880108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:13.077 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:11 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3257164987' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.820+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34680a2800 msgr2=0x7f34680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34680a2800 0x7f34680a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f347c101750 tx=0x7f346c005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1022b0 msgr2=0x7f347c19ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1022b0 0x7f347c19ed70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f347400eb10 tx=0x7f347400c2d0 comp rx=0 tx=0).stop 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 shutdown_connections 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34680a2800 0x7f34680a4cc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f347c1022b0 0x7f347c19ed70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f347c1018f0 0x7f347c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 --2- 172.21.15.77:0/3257164987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f347c100700 0x7f347c06a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 >> 172.21.15.77:0/3257164987 conn(0x7f347c0fbc90 msgr2=0x7f347c104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 shutdown_connections 2024-03-16T09:45:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:11.821+0000 7f34817c6640 1 -- 172.21.15.77:0/3257164987 wait complete. 2024-03-16T09:45:13.084 DEBUG:tasks.check_counter:Getting stats from a 2024-03-16T09:45:13.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-16T09:45:13.085 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:11 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2442420345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:13.085 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:11 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/921880108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:13.085 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:11 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3257164987' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T09:45:13.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:12 smithi119 ceph-mon[35632]: pgmap v445: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.3 KiB/s wr, 187 op/s 2024-03-16T09:45:13.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:12 smithi077 ceph-mon[32171]: pgmap v445: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.3 KiB/s wr, 187 op/s 2024-03-16T09:45:13.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:12 smithi129 ceph-mon[35841]: pgmap v445: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.3 KiB/s wr, 187 op/s 2024-03-16T09:45:13.336 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:45:15.092 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:14 smithi077 ceph-mon[32171]: pgmap v446: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 74 op/s 2024-03-16T09:45:15.092 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:45:15.092 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:15.092 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 4, 2024-03-16T09:45:15.092 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 4, 2024-03-16T09:45:15.092 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 2992, 2024-03-16T09:45:15.092 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 448, 2024-03-16T09:45:15.092 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001461994, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000033560, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000539098, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000737672, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000245890 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:15.093 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 94, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 91, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 440512, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 54752, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.013929652, 2024-03-16T09:45:15.094 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.004960504, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.008735535, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.001798120, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 90, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.003884168, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000043157 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 440512, 2024-03-16T09:45:15.095 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 54752 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 69, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 160, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 89808, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 3, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.005761568, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.003639687, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001438605, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:15.096 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 67, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.003384697, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000050517 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 89808 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-03-16T09:45:15.097 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:15.098 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:15.099 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148823, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 27, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T09:45:15.100 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 95, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 427272, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 95, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 95, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 427272, 2024-03-16T09:45:15.102 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T09:45:15.103 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:15.103 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:15.103 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:15.103 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:15.103 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:15.103 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:45:15.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:14 smithi129 ceph-mon[35841]: pgmap v446: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 74 op/s 2024-03-16T09:45:15.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:14 smithi119 ceph-mon[35632]: pgmap v446: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 74 op/s 2024-03-16T09:45:15.882 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-16T09:45:15.882 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-16T09:45:15.882 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-16T09:45:15.882 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-16T09:45:15.882 DEBUG:tasks.check_counter:Getting stats from d 2024-03-16T09:45:15.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-16T09:45:16.145 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:45:17.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:16 smithi077 ceph-mon[32171]: pgmap v447: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-16T09:45:17.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:16 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:45:17.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:16 smithi129 ceph-mon[35841]: pgmap v447: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-16T09:45:17.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:16 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:45:17.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:16 smithi119 ceph-mon[35632]: pgmap v447: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-16T09:45:17.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:16 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 0, 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2, 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 0, 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 0, 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001051768, 2024-03-16T09:45:17.917 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000216129, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:17.918 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 73, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 160, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 90512, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 3, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.006160024, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.004235167, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001184128, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:17.919 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 71, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.003632824, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000051166 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 90512 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 104, 2024-03-16T09:45:17.920 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 99, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 445136, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 59040, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.015654227, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.005435440, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.010698371, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.001812388, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 99, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.004691037, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000047384 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.921 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 445136, 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 59040 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T09:45:17.922 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:17.926 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:17.927 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:17.928 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148779, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 27, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:17.929 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 101, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 428052, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 101, 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 101, 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 428052, 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:17.931 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:45:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:17 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:45:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:17 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:45:18.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:17 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:45:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:17 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:45:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:17 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:45:18.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:17 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:45:18.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:17 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T09:45:18.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:17 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T09:45:18.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:17 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' 2024-03-16T09:45:18.644 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-16T09:45:18.644 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-16T09:45:18.645 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-16T09:45:18.645 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-16T09:45:18.645 DEBUG:tasks.check_counter:Getting stats from g 2024-03-16T09:45:18.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-16T09:45:18.890 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:45:19.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:18 smithi077 ceph-mon[32171]: pgmap v448: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:45:19.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:18 smithi129 ceph-mon[35841]: pgmap v448: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:45:19.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:18 smithi119 ceph-mon[35632]: pgmap v448: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T09:45:20.653 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:45:20.653 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:20.653 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 0, 2024-03-16T09:45:20.653 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2, 2024-03-16T09:45:20.653 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 0, 2024-03-16T09:45:20.653 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 0, 2024-03-16T09:45:20.653 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001177983, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000263946, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:20.654 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1, 2024-03-16T09:45:20.655 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 70, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 160, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 89984, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 3, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.005790213, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.003854851, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001204405, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 68, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.003317340, 2024-03-16T09:45:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000048784 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 89984 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 100, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 95, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 444048, 2024-03-16T09:45:20.657 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 56480, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.015167345, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.005689176, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.009457386, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.001677900, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 95, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.005019199, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000052833 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:20.658 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 444048, 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 56480 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T09:45:20.659 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:20.660 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:20.661 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:20.662 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148823, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 27, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-03-16T09:45:20.663 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:20.664 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:20.664 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:20.664 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T09:45:20.664 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-03-16T09:45:20.664 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-03-16T09:45:20.664 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 97, 2024-03-16T09:45:20.665 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 427532, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 97, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 97, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 427532, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:20.666 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:45:21.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:20 smithi077 ceph-mon[32171]: pgmap v449: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-16T09:45:21.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:20 smithi129 ceph-mon[35841]: pgmap v449: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-16T09:45:21.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:20 smithi119 ceph-mon[35632]: pgmap v449: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-16T09:45:21.472 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-16T09:45:21.472 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-16T09:45:21.472 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-16T09:45:21.473 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-16T09:45:21.473 DEBUG:tasks.check_counter:Getting stats from j 2024-03-16T09:45:21.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-16T09:45:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T09:45:23.236 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:22 smithi077 ceph-mon[32171]: pgmap v450: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:45:23.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:22 smithi129 ceph-mon[35841]: pgmap v450: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:45:23.442 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:22 smithi119 ceph-mon[35632]: pgmap v450: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T09:45:23.504 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T09:45:23.504 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:23.504 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 0, 2024-03-16T09:45:23.504 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2, 2024-03-16T09:45:23.504 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 0, 2024-03-16T09:45:23.504 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 0, 2024-03-16T09:45:23.504 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001094505, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000280247, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:23.505 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 5, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 77, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 3152, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 91312, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 3, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.006919515, 2024-03-16T09:45:23.506 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.004181129, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.002096408, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 76, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.004604179, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000060581 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 91312 2024-03-16T09:45:23.507 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 102, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 99, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 442688, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 59872, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.016387010, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.005615010, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.011213051, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.001952349, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 98, 2024-03-16T09:45:23.508 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.004648595, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000047434 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 442688, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 59872 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-03-16T09:45:23.509 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:23.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:23.511 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148823, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 27, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T09:45:23.512 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T09:45:23.513 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 103, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 428312, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 103, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T09:45:23.514 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 103, 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 428312, 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T09:45:23.515 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T09:45:24.169 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-16T09:45:24.169 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-16T09:45:24.169 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-16T09:45:24.169 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-16T09:45:24.169 DEBUG:tasks.check_counter:Getting stats from b 2024-03-16T09:45:24.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-16T09:45:24.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:23 smithi077 ceph-mon[32171]: pgmap v451: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:24.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:23 smithi129 ceph-mon[35841]: pgmap v451: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:24.427 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:45:24.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:23 smithi119 ceph-mon[35632]: pgmap v451: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:26.387 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-16T09:45:26.387 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:26.387 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1525, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1464, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 848113, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 1356669, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 82, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 6, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.124794252, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.033262843, 2024-03-16T09:45:26.388 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.055867938, 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.028335995, 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1467, 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.364059843, 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000248166 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 529, 2024-03-16T09:45:26.389 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000088169, 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000166 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 482336, 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 70320 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1964, 2024-03-16T09:45:26.390 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1813, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 569467, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 848410, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 12, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.120078787, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.032004461, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.039451950, 2024-03-16T09:45:26.391 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.043528132, 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1811, 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.402725770, 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000222377 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 674, 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000099635, 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000147 2024-03-16T09:45:26.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 57280, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 237664 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1722, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1687, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 463721, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 1005014, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 11, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.112054576, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.033630458, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.033128937, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.035054404, 2024-03-16T09:45:26.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1682, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.401802617, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000238883 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 447, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000074871, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000167 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.394 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-16T09:45:26.395 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-16T09:45:26.395 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-16T09:45:26.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.395 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-16T09:45:26.395 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-16T09:45:26.395 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-16T09:45:26.395 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 485, 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.125129529, 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000257999 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2845, 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.100968110, 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000035489 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.396 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 50, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 50, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 50, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.066685724, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001333714 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 0, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 78, 2024-03-16T09:45:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 23, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 4, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 19, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 16, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 0, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 3, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 3, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 2, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 74, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 66, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 0, 2024-03-16T09:45:26.398 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 21, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 78, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 4, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 234, 2024-03-16T09:45:26.399 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 78, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 51, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 6, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 45, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 27, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 18, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 74, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T09:45:26.400 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 82, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 59, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 5, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 32 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 6, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 0, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 14, 2024-03-16T09:45:26.401 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-16T09:45:26.402 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-16T09:45:26.402 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-16T09:45:26.402 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 5, 2024-03-16T09:45:26.402 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 5, 2024-03-16T09:45:26.402 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-16T09:45:26.402 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-16T09:45:26.402 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 6, 2024-03-16T09:45:26.403 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 142, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 132, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 132, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 10, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 2, 2024-03-16T09:45:26.404 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 15, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 14, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 14, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 1, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 1, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 1, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4570673, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 4600170, 2024-03-16T09:45:26.405 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 121, 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.588419130, 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004862968 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 23, 2024-03-16T09:45:26.406 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 34, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 11, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 18, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 20, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 2, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 23, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 32, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 9, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 3, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 19, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 16, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 55384, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 231804 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-16T09:45:26.407 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 73, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 50, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 11, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 7, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.008437381, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001205340 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 17, 2024-03-16T09:45:26.408 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.005559081, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000327004 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.409 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.412 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-16T09:45:26.412 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.412 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.412 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 8, 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.012373390, 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001546673 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.413 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1, 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.001097461, 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001097461 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.414 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.003259764, 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001629882 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.003682623, 2024-03-16T09:45:26.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001841311 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.008899948, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004449974 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.417 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1, 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.001360573, 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001360573 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.418 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-16T09:45:26.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 7, 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.016361646, 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002337378 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.420 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.005653857, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001884619 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-16T09:45:26.421 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 2, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 13, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 13, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 218, 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-16T09:45:26.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:26.423 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:26.424 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:26.425 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 299841, 2024-03-16T09:45:26.426 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 122, 2024-03-16T09:45:26.426 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 88, 2024-03-16T09:45:26.426 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 1, 2024-03-16T09:45:26.426 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-16T09:45:26.426 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-16T09:45:26.426 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:26.426 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 7528, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 158, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:26.427 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 94986, 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 657, 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.428 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 4018, 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 493585, 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 36, 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 4018, 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-16T09:45:26.429 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4018, 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 33.406163703, 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.008314127 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4018, 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 4234 2024-03-16T09:45:26.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 4018, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 95, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 3923, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 58, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 31, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 11, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 126, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 58, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-03-16T09:45:26.431 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 3586, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 84, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 62, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-16T09:45:26.432 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-16T09:45:26.433 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 88, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 3, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 200, 2024-03-16T09:45:26.434 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 18 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.435 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 2560, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 1, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 3074, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 5, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 5207, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1395693, 2024-03-16T09:45:26.436 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 5207, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 5207, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1395693, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.437 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 4018, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 494090, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 4018, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 494090, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.438 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 4018, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 4018, 2024-03-16T09:45:26.439 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 4018, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 4018, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 5, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 505, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 5, 2024-03-16T09:45:26.440 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 5, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 505, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x560b17068500": { 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 142, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 405866, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:26.441 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 142, 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 121, 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 405866, 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:26.442 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-16T09:45:27.085 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:26 smithi119 ceph-mon[35632]: pgmap v452: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:27.087 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-16T09:45:27.087 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-16T09:45:27.087 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=6 2024-03-16T09:45:27.087 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-16T09:45:27.087 DEBUG:tasks.check_counter:Getting stats from e 2024-03-16T09:45:27.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-16T09:45:27.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:26 smithi077 ceph-mon[32171]: pgmap v452: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:27.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:26 smithi129 ceph-mon[35841]: pgmap v452: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:27.390 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:45:29.114 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:28 smithi119 ceph-mon[35632]: pgmap v453: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:29.114 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-16T09:45:29.114 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:29.114 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 829, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 823, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1081226, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 288720, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.060037115, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.020462777, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.021630923, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.016449795, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 823, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.019473244, 2024-03-16T09:45:29.115 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000023661 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 464256, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 61664 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 5, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 79, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1816, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 208138, 2024-03-16T09:45:29.116 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.004384479, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.002400432, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.001343764, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000090983, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 77, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.005526191, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000071768 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 206800 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 316, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 318, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 77842, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 100495, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.021075665, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.005930107, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.004945261, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.008362072, 2024-03-16T09:45:29.118 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 316, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.008524790, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000026977 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-16T09:45:29.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1735, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.036721597, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000021165 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-16T09:45:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 21, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 6, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 15, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 12, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 2, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-16T09:45:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 4, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 234, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T09:45:29.123 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 0, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 0 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.124 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-16T09:45:29.125 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 0, 2024-03-16T09:45:29.126 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 142, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 2, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 15, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4570673, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 4600170, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 4596412, 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-16T09:45:29.127 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 142 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 16, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 21, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 5, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 11, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 13, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 2, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 21, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 25, 2024-03-16T09:45:29.128 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 4, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 49528, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 215420 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.130 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-16T09:45:29.131 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.133 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-16T09:45:29.133 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.133 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.133 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.134 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-16T09:45:29.137 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-16T09:45:29.138 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.139 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-16T09:45:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 2, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 0, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 0, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 1260, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:29.141 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:29.142 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:29.143 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:29.144 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 189279, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 49, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:29.145 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 7528, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 158, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 64394, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 296, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-16T09:45:29.146 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 1034, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 26, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 4, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 1034, 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-16T09:45:29.147 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1034, 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.444988830, 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000430356 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1034, 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1035 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 1034, 2024-03-16T09:45:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 1034, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 642, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 389, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-16T09:45:29.149 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 2, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T09:45:29.150 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-16T09:45:29.151 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 88, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 2, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 5, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 5, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 0, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 2, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 0 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-16T09:45:29.152 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 0, 2024-03-16T09:45:29.153 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 1146, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1049061, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 1146, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 1146, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1049061, 2024-03-16T09:45:29.154 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:29.155 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 1034, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 405876, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 1034, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 405876, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:29.156 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 1034, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 1034, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 1034, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1034, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:29.157 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.158 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x55fa06a0c6e0": { 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-16T09:45:29.159 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-16T09:45:29.160 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:29.160 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:29.160 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:29.160 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:29.160 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.160 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:29.160 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-16T09:45:29.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:28 smithi077 ceph-mon[32171]: pgmap v453: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:29.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:28 smithi129 ceph-mon[35841]: pgmap v453: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:29.853 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-16T09:45:29.854 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-16T09:45:29.854 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-16T09:45:29.854 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-16T09:45:29.854 DEBUG:tasks.check_counter:Getting stats from h 2024-03-16T09:45:29.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-16T09:45:30.146 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:45:31.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:30 smithi119 ceph-mon[35632]: pgmap v454: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:31.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:30 smithi077 ceph-mon[32171]: pgmap v454: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:31.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:30 smithi129 ceph-mon[35841]: pgmap v454: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 240, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 290, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 67373, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 234332, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 21, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.025257195, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.009292799, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.013081144, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.001555872, 2024-03-16T09:45:31.740 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 288, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.032652077, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000113375 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 461, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000082542, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000179 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:31.741 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 166, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 158, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 480260, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 119125, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 11, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.017205325, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.004476812, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.014779512, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.002073190, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 157, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.011313458, 2024-03-16T09:45:31.742 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000072060 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000002867, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000130 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 474800, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 65216 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 32, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 103, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 14866, 2024-03-16T09:45:31.743 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 227461, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 7, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.006496854, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.002942620, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.003091966, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000193438, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 102, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.012576089, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000123294 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 28, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000004981, 2024-03-16T09:45:31.744 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000177 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 208688 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 36, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.001868049, 2024-03-16T09:45:31.745 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000051890 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000006539, 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000003269 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 4, 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 2, 2024-03-16T09:45:31.746 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000722527, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000361263 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 2, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 11, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 16, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 0, 2024-03-16T09:45:31.747 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 16, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 13, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 2, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 2, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 2, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 3, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 10, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 5, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 2, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 3, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-16T09:45:31.748 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 5, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 10737418474, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T09:45:31.749 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 0 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 0, 2024-03-16T09:45:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 9, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 6, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 7, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-16T09:45:31.751 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 3, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 3, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-16T09:45:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 2, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 2, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 2, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 4195385, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.002929856, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001464928 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-03-16T09:45:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 20, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 21, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 1, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 19, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 21, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 2, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 16, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 18, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 2, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 2, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 2, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 0, 2024-03-16T09:45:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 54208, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 223612 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 4, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 4, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 5, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-16T09:45:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000722527, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000361263 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-16T09:45:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.759 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.761 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-16T09:45:31.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.763 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 1, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 0, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 2, 2024-03-16T09:45:31.765 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 2, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 222, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:31.766 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:31.767 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 254619, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 82, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 7528, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 158, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 81373, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 330, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:31.769 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 38, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 11452, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 33, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 38, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 38, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.230994223, 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.006078795 2024-03-16T09:45:31.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 38, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 61 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 38, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 0, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 38, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 10, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 0, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 2, 2024-03-16T09:45:31.771 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 6, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 6, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 20, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-16T09:45:31.772 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-16T09:45:31.773 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 88, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 2, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 9, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 9, 2024-03-16T09:45:31.774 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 16, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 1 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-16T09:45:31.775 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 0, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 0, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 0, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.776 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 434, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 513288, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 434, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 434, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 513288, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.777 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 38, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 11452, 2024-03-16T09:45:31.778 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 38, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 11452, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:31.779 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 38, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 38, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 38, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 38, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-16T09:45:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x55d76f738500": { 2024-03-16T09:45:31.781 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 3, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1081, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 3, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1081, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:31.782 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.783 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:31.783 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-16T09:45:32.536 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-16T09:45:32.536 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-16T09:45:32.536 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-16T09:45:32.536 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=9 2024-03-16T09:45:32.536 DEBUG:tasks.check_counter:Getting stats from k 2024-03-16T09:45:32.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-16T09:45:32.786 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T09:45:33.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:32 smithi119 ceph-mon[35632]: pgmap v455: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:45:33.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:32 smithi077 ceph-mon[32171]: pgmap v455: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:45:33.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:32 smithi129 ceph-mon[35841]: pgmap v455: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 0, 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 2, 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 0, 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 0, 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:34.381 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 0, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.001073919, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.000308250, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:34.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 78, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 160, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 91392, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 3, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.003959861, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.002310738, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.001079941, 2024-03-16T09:45:34.383 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 76, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.002743966, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000036104 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 91392 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:34.384 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 110, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 105, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 447392, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 62576, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 2, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.014275877, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.004164234, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.010408782, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.001922914, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 105, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.004103237, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000039078 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.385 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 447392, 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 62576 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 0, 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-16T09:45:34.386 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:34.388 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 148863, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 28, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:34.389 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 928, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 2, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-16T09:45:34.390 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 107, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 429469, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 107, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 107, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 429469, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-16T09:45:34.391 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-16T09:45:34.392 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:34.392 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-16T09:45:34.392 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-16T09:45:35.042 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:34 smithi119 ceph-mon[35632]: pgmap v456: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:35.070 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-16T09:45:35.070 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-16T09:45:35.070 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-16T09:45:35.071 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-16T09:45:35.071 DEBUG:tasks.check_counter:Getting stats from c 2024-03-16T09:45:35.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-16T09:45:35.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:34 smithi077 ceph-mon[32171]: pgmap v456: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:35.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:34 smithi129 ceph-mon[35841]: pgmap v456: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:45:37.006 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:36 smithi129 ceph-mon[35841]: pgmap v457: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:37.007 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 449, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 445, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 575504, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 169198, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.038123023, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.011146266, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.016793216, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.009874215, 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:37.008 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 444, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.015740054, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000035450 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 462272, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 62032 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 7, 2024-03-16T09:45:37.009 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 79, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 3708, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 209562, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.004510681, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.002227050, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.001739538, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000071241, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 78, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.004660096, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000059744 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.010 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 208864 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 571, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 573, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 147584, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 181585, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-03-16T09:45:37.011 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.029956591, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.009049015, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.008119773, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.010866702, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 571, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.014019861, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000024553 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1584, 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.031444894, 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000019851 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.013 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 0, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.014 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 0, 2024-03-16T09:45:37.015 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 1, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 0, 2024-03-16T09:45:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T09:45:37.017 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 0 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-03-16T09:45:37.018 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-03-16T09:45:37.019 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 0, 2024-03-16T09:45:37.020 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 3, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 0, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 0, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 0, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 0, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 2, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 2, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 0, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 4194304, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 4195385, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 4194892, 2024-03-16T09:45:37.021 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 3 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 3, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 3, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 0, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 1, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 1, 2024-03-16T09:45:37.022 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 49548, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 215420 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 0, 2024-03-16T09:45:37.023 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 0, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.025 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.026 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-03-16T09:45:37.027 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.029 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.031 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.032 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.033 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-03-16T09:45:37.034 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 1, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 1, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 0, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 1, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 0, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 0, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 1356, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:37.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:37.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:37.037 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 181137, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 44, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 7528, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 158, 2024-03-16T09:45:37.038 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 8464, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 13, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-03-16T09:45:37.039 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 910, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 21, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 3, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 910, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 910, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.621986267, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000683501 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 910, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 911 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 910, 2024-03-16T09:45:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 910, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 566, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 342, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-03-16T09:45:37.041 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 1, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-03-16T09:45:37.042 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 88, 2024-03-16T09:45:37.043 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 2, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 4, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 4, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 0, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 2, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 0 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 0, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-03-16T09:45:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "osd-slow-ops": { 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 0 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 0, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 0, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 0, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:37.045 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 1023, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 625998, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 1023, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1023, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 625998, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.046 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 910, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 1086, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 910, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1086, 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:37.047 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 910, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 910, 2024-03-16T09:45:37.048 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 910, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 910, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:37.049 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x55a20b3b06e0": { 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:37.050 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:37.051 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-16T09:45:37.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:36 smithi077 ceph-mon[32171]: pgmap v457: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:37.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:36 smithi119 ceph-mon[35632]: pgmap v457: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:37.796 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-16T09:45:37.796 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-16T09:45:37.796 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-16T09:45:37.796 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-16T09:45:37.796 DEBUG:tasks.check_counter:Getting stats from f 2024-03-16T09:45:37.797 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-16T09:45:38.057 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:45:39.158 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:38 smithi129 ceph-mon[35841]: pgmap v458: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:39.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:38 smithi077 ceph-mon[32171]: pgmap v458: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:39.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:38 smithi119 ceph-mon[35632]: pgmap v458: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 345, 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 347, 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 92078, 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 109774, 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 6, 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:39.749 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.021888518, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.006261610, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.006468532, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.006394019, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 345, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.010116205, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000029322 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:39.750 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 239, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 312, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 60738, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 285073, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 4, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.019863227, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.006219284, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.005676703, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.006205430, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 311, 2024-03-16T09:45:39.751 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.010444543, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000033583 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 210656 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 453, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 449, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 576784, 2024-03-16T09:45:39.752 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 170792, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 3, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.037620923, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.011094975, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.017048833, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.009771331, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 448, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.014504952, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000032377 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.753 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 462544, 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 62672 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-03-16T09:45:39.754 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1551, 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.030651648, 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000019762 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-03-16T09:45:39.755 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-03-16T09:45:39.756 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 1, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-03-16T09:45:39.757 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-03-16T09:45:39.758 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 0, 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 0 2024-03-16T09:45:39.759 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-03-16T09:45:39.760 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 0, 2024-03-16T09:45:39.761 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 3, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 2, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 2, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 4194304, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 4195385, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 4194892, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.762 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 3 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 3, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 3, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 0, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 1, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 1, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 0, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 0, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 0, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 0, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 0, 2024-03-16T09:45:39.763 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 49416, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 215420 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.764 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.765 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.766 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-03-16T09:45:39.767 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-03-16T09:45:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.769 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.770 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.771 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.772 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.773 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-03-16T09:45:39.774 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 1, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 1, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 1, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 1358, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:39.775 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:39.776 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 189279, 2024-03-16T09:45:39.777 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 49, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 7528, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 158, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 8464, 2024-03-16T09:45:39.778 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 13, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 919, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 21, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 3, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 919, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 919, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.471989558, 2024-03-16T09:45:39.779 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000513590 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 919, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 920 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 919, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 919, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 0, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 572, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 0, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 345, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 0, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 0, 2024-03-16T09:45:39.780 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 1, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T09:45:39.781 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-03-16T09:45:39.782 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 88, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 2, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 5, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 5, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-03-16T09:45:39.783 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 0, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 2, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 0 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 0, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "osd-slow-ops": { 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 0 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-03-16T09:45:39.784 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 1033, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 627850, 2024-03-16T09:45:39.785 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 1033, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 1033, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 627850, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-03-16T09:45:39.786 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 919, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 1086, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 919, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1086, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.787 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 919, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 919, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 919, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 919, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.788 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-03-16T09:45:39.789 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x562e21bfe6e0": { 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:39.790 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:39.791 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-16T09:45:40.391 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-16T09:45:40.391 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-16T09:45:40.391 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-16T09:45:40.391 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-16T09:45:40.391 DEBUG:tasks.check_counter:Getting stats from i 2024-03-16T09:45:40.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-16T09:45:40.637 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:45:41.014 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:40 smithi129 ceph-mon[35841]: pgmap v459: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:41.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:40 smithi119 ceph-mon[35632]: pgmap v459: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:41.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:40 smithi077 ceph-mon[32171]: pgmap v459: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:42.308 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-16T09:45:42.308 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 31, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 35, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 9392, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 57413, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 13, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 4, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.005340821, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000377063, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.002907135, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000337786, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 30, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.013728493, 2024-03-16T09:45:42.309 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000457616 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 14, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000001477, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000105 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 27, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 103, 2024-03-16T09:45:42.310 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 5684, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 361126, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 19, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 5, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.010428697, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.002612094, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.006737817, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000335275, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 99, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.017343714, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000175189 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:42.311 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 12, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000001345, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000112 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 210032 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 374, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 406, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 543134, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 163286, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 6, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-03-16T09:45:42.312 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.031328987, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.010818722, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.020950045, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.003061713, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 411, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.017380093, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000042287 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 491, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000076207, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000155 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 470576, 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 65696 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 30, 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.001498886, 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000049962 2024-03-16T09:45:42.314 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 2, 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000009309, 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000004654 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 0, 2024-03-16T09:45:42.315 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 11, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 16, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 0, 2024-03-16T09:45:42.316 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 16, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 0, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 13, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 2, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 0, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 0, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 3, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 6, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 3, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 0, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 3, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-03-16T09:45:42.317 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 1000, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 1, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 5, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 10737418474, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T09:45:42.318 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 0, 2024-03-16T09:45:42.319 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 0 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 9, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 6, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 7, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-03-16T09:45:42.320 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-03-16T09:45:42.321 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 3, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 3, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 2, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 2, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 2, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 0, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 4194304, 2024-03-16T09:45:42.322 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 4195385, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 0, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 2, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.002734385, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001367192 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 0 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 20, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 21, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 1, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 19, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 21, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 2, 2024-03-16T09:45:42.323 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 16, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 18, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 2, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 0, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 0, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 0, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 53828, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 223612 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 0, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 0, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 0, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 5, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 0, 2024-03-16T09:45:42.324 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.325 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.326 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.327 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-03-16T09:45:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.329 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.330 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.332 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-03-16T09:45:42.333 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.334 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 1, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 1, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 0, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 1, 2024-03-16T09:45:42.335 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 230, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:42.337 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 221655, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 65, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:42.338 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 7528, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 158, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 80597, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 321, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-03-16T09:45:42.339 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 37, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 11293, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 28, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 37, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 37, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.198995587, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.005378259 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-03-16T09:45:42.340 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 37, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 59 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 37, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 0, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 37, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 0, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 10, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 0, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 2, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 6, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-03-16T09:45:42.341 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 6, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 20, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-03-16T09:45:42.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-03-16T09:45:42.343 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 88, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 2, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 8, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 8, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 14, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 0, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 1 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 1, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-03-16T09:45:42.344 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "osd-slow-ops": { 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 0 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 0, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 0, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 0, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.345 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 428, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 509755, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 428, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 428, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 509755, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.346 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 37, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 11293, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 37, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 11293, 2024-03-16T09:45:42.347 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 37, 2024-03-16T09:45:42.348 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 37, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 37, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 37, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 0, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 0, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x564b8d108500": { 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 3, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1081, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 3, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 2, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1081, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:42.351 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-16T09:45:43.046 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-16T09:45:43.046 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-16T09:45:43.046 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-16T09:45:43.046 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=9 2024-03-16T09:45:43.046 DEBUG:tasks.check_counter:Getting stats from l 2024-03-16T09:45:43.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid e287e1d6-e377-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-16T09:45:43.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:42 smithi077 ceph-mon[32171]: pgmap v460: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:45:43.296 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T09:45:43.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:42 smithi129 ceph-mon[35841]: pgmap v460: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:45:43.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:42 smithi119 ceph-mon[35632]: pgmap v460: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T09:45:45.027 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:44 smithi129 ceph-mon[35841]: pgmap v461: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:45.027 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 0, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 2, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 0, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 0, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.001043640, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000256835, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:45.028 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 114, 2024-03-16T09:45:45.029 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 109, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 448480, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 65136, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.014037950, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.004243722, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.010197229, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.001832364, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 109, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.004426036, 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000040605 2024-03-16T09:45:45.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 448480, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 65136 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 1, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 80, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 160, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 91744, 2024-03-16T09:45:45.031 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 3, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 2, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.003384440, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.002153580, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000629259, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 78, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.003106841, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000039831 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:45.032 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 91744 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-03-16T09:45:45.033 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-03-16T09:45:45.034 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-03-16T09:45:45.035 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 148903, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 29, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-03-16T09:45:45.036 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-03-16T09:45:45.037 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 111, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 429989, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 111, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 111, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 429989, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-16T09:45:45.038 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-16T09:45:45.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:44 smithi077 ceph-mon[32171]: pgmap v461: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:45.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:44 smithi119 ceph-mon[35632]: pgmap v461: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:45.778 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-16T09:45:45.778 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-16T09:45:45.778 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-16T09:45:45.778 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-16T09:45:45.778 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-16T09:45:46.018 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-16T09:45:46.018 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:45:46.018 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T09:45:46.040 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:45:46.040 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T09:45:46.101 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 -- 172.21.15.77:0/3049016145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc107460 msgr2=0x7feffc109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 --2- 172.21.15.77:0/3049016145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc107460 0x7feffc109920 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7feff4009ca0 tx=0x7feff402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 -- 172.21.15.77:0/3049016145 shutdown_connections 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 --2- 172.21.15.77:0/3049016145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc107460 0x7feffc109920 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 --2- 172.21.15.77:0/3049016145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feffc0ffed0 0x7feffc106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 --2- 172.21.15.77:0/3049016145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feffc0ff570 0x7feffc0ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 -- 172.21.15.77:0/3049016145 >> 172.21.15.77:0/3049016145 conn(0x7feffc0fafe0 msgr2=0x7feffc0fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:46.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 -- 172.21.15.77:0/3049016145 shutdown_connections 2024-03-16T09:45:46.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.255+0000 7ff0011d4640 1 -- 172.21.15.77:0/3049016145 wait complete. 2024-03-16T09:45:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.256+0000 7ff0011d4640 1 Processor -- start 2024-03-16T09:45:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7ff0011d4640 1 -- start start 2024-03-16T09:45:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7ff0011d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc0ff570 0x7feffc1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7ff0011d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feffc0ffed0 0x7feffc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7ff0011d4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feffc107460 0x7feffc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7ff0011d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feffc10d320 con 0x7feffc0ff570 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7ff0011d4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feffc10d1a0 con 0x7feffc107460 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7ff0011d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feffc10d4a0 con 0x7feffc0ffed0 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc0ff570 0x7feffc1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc0ff570 0x7feffc1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42364/0 (socket says 172.21.15.77:42364) 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 -- 172.21.15.77:0/2382359326 learned_addr learned my addr 172.21.15.77:0/2382359326 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffb59e640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feffc107460 0x7feffc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffa59c640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feffc0ffed0 0x7feffc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 -- 172.21.15.77:0/2382359326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feffc0ffed0 msgr2=0x7feffc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feffc0ffed0 0x7feffc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 -- 172.21.15.77:0/2382359326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feffc107460 msgr2=0x7feffc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feffc107460 0x7feffc19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.257+0000 7feffad9d640 1 -- 172.21.15.77:0/2382359326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc19f610 con 0x7feffc0ff570 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.258+0000 7feffad9d640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc0ff570 0x7feffc1979f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fefec00e9f0 tx=0x7fefec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.258+0000 7fefe3fff640 1 -- 172.21.15.77:0/2382359326 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec00cd80 con 0x7feffc0ff570 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.258+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc19f900 con 0x7feffc0ff570 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.258+0000 7fefe3fff640 1 -- 172.21.15.77:0/2382359326 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefec0108e0 con 0x7feffc0ff570 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.258+0000 7fefe3fff640 1 -- 172.21.15.77:0/2382359326 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec018920 con 0x7feffc0ff570 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.258+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc19fea0 con 0x7feffc0ff570 2024-03-16T09:45:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.259+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feffc105ce0 con 0x7feffc0ff570 2024-03-16T09:45:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.260+0000 7fefe3fff640 1 -- 172.21.15.77:0/2382359326 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fefec010a90 con 0x7feffc0ff570 2024-03-16T09:45:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.261+0000 7fefe3fff640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fefe40a2800 0x7fefe40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.261+0000 7feffa59c640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fefe40a2800 0x7fefe40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.261+0000 7fefe3fff640 1 -- 172.21.15.77:0/2382359326 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fefec0ced40 con 0x7feffc0ff570 2024-03-16T09:45:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.261+0000 7feffa59c640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fefe40a2800 0x7fefe40a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feffc198f00 tx=0x7feff0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:46.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.265+0000 7fefe3fff640 1 -- 172.21.15.77:0/2382359326 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fefec09b7a0 con 0x7feffc0ff570 2024-03-16T09:45:46.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.443+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feffc0623e0 con 0x7feffc0ff570 2024-03-16T09:45:46.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.445+0000 7fefe3fff640 1 -- 172.21.15.77:0/2382359326 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7fefec09f840 con 0x7feffc0ff570 2024-03-16T09:45:46.448 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:46.448 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:45:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fefe40a2800 msgr2=0x7fefe40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fefe40a2800 0x7fefe40a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feffc198f00 tx=0x7feff0005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc0ff570 msgr2=0x7feffc1979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc0ff570 0x7feffc1979f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fefec00e9f0 tx=0x7fefec00eeb0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 shutdown_connections 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fefe40a2800 0x7fefe40a4cc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feffc107460 0x7feffc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feffc0ffed0 0x7feffc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 --2- 172.21.15.77:0/2382359326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feffc0ff570 0x7feffc1979f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 >> 172.21.15.77:0/2382359326 conn(0x7feffc0fafe0 msgr2=0x7feffc102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 shutdown_connections 2024-03-16T09:45:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.447+0000 7ff0011d4640 1 -- 172.21.15.77:0/2382359326 wait complete. 2024-03-16T09:45:46.462 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:45:46.463 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:45:46.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 -- 172.21.15.77:0/3562685370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c105370 msgr2=0x7f987c1057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 --2- 172.21.15.77:0/3562685370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c105370 0x7f987c1057f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9874009ca0 tx=0x7f987402c0f0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 -- 172.21.15.77:0/3562685370 shutdown_connections 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 --2- 172.21.15.77:0/3562685370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c105370 0x7f987c1057f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 --2- 172.21.15.77:0/3562685370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f987c1029c0 0x7f987c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 --2- 172.21.15.77:0/3562685370 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f987c100050 0x7f987c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 -- 172.21.15.77:0/3562685370 >> 172.21.15.77:0/3562685370 conn(0x7f987c0fbc70 msgr2=0x7f987c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 -- 172.21.15.77:0/3562685370 shutdown_connections 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 -- 172.21.15.77:0/3562685370 wait complete. 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.564+0000 7f9883f6f640 1 Processor -- start 2024-03-16T09:45:46.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 -- start start 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f987c100050 0x7f987c19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c1029c0 0x7f987c19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f987c105370 0x7f987c1a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f987c113900 con 0x7f987c1029c0 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f987c113780 con 0x7f987c105370 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f987c113a80 con 0x7f987c100050 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c1029c0 0x7f987c19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c1029c0 0x7f987c19e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42374/0 (socket says 172.21.15.77:42374) 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 -- 172.21.15.77:0/2839677642 learned_addr learned my addr 172.21.15.77:0/2839677642 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9881d0b640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f987c100050 0x7f987c19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988250c640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f987c105370 0x7f987c1a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 -- 172.21.15.77:0/2839677642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f987c100050 msgr2=0x7f987c19e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f987c100050 0x7f987c19e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 -- 172.21.15.77:0/2839677642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f987c105370 msgr2=0x7f987c1a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f987c105370 0x7f987c1a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 -- 172.21.15.77:0/2839677642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c1a5c30 con 0x7f987c1029c0 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f988150a640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c1029c0 0x7f987c19e550 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f986c00e9f0 tx=0x7f986c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9872ffd640 1 -- 172.21.15.77:0/2839677642 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c00cd80 con 0x7f987c1029c0 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9872ffd640 1 -- 172.21.15.77:0/2839677642 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f986c0054d0 con 0x7f987c1029c0 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987c1a5f20 con 0x7f987c1029c0 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.565+0000 7f9872ffd640 1 -- 172.21.15.77:0/2839677642 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c010490 con 0x7f987c1029c0 2024-03-16T09:45:46.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.566+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987c1a6390 con 0x7f987c1029c0 2024-03-16T09:45:46.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.566+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f987c0fd760 con 0x7f987c1029c0 2024-03-16T09:45:46.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.568+0000 7f9872ffd640 1 -- 172.21.15.77:0/2839677642 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f986c005000 con 0x7f987c1029c0 2024-03-16T09:45:46.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.568+0000 7f9872ffd640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f98680a2800 0x7f98680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.568+0000 7f9881d0b640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f98680a2800 0x7f98680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.568+0000 7f9872ffd640 1 -- 172.21.15.77:0/2839677642 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f986c0cefa0 con 0x7f987c1029c0 2024-03-16T09:45:46.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.569+0000 7f9881d0b640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f98680a2800 0x7f98680a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9878009bd0 tx=0x7f9878005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:46.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.571+0000 7f9872ffd640 1 -- 172.21.15.77:0/2839677642 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f986c053740 con 0x7f987c1029c0 2024-03-16T09:45:46.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.745+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f987c02d340 con 0x7f987c1029c0 2024-03-16T09:45:46.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.746+0000 7f9872ffd640 1 -- 172.21.15.77:0/2839677642 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f986c0052f0 con 0x7f987c1029c0 2024-03-16T09:45:46.748 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.747+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f98680a2800 msgr2=0x7f98680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.747+0000 7f9883f6f640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f98680a2800 0x7f98680a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9878009bd0 tx=0x7f9878005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.747+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c1029c0 msgr2=0x7f987c19e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.747+0000 7f9883f6f640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c1029c0 0x7f987c19e550 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f986c00e9f0 tx=0x7f986c00edf0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 shutdown_connections 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f98680a2800 0x7f98680a4cc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f987c105370 0x7f987c1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f987c1029c0 0x7f987c19e550 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 --2- 172.21.15.77:0/2839677642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f987c100050 0x7f987c19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 >> 172.21.15.77:0/2839677642 conn(0x7f987c0fbc70 msgr2=0x7f987c1009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 shutdown_connections 2024-03-16T09:45:46.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.748+0000 7f9883f6f640 1 -- 172.21.15.77:0/2839677642 wait complete. 2024-03-16T09:45:46.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:45:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 -- 172.21.15.77:0/88903202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1588102290 msgr2=0x7f1588108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 --2- 172.21.15.77:0/88903202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1588102290 0x7f1588108b20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1580009ca0 tx=0x7f158002c0f0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 -- 172.21.15.77:0/88903202 shutdown_connections 2024-03-16T09:45:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 --2- 172.21.15.77:0/88903202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1588102290 0x7f1588108b20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 --2- 172.21.15.77:0/88903202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15881018d0 0x7f1588101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 --2- 172.21.15.77:0/88903202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15881006e0 0x7f1588100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 -- 172.21.15.77:0/88903202 >> 172.21.15.77:0/88903202 conn(0x7f15880fbc70 msgr2=0x7f15880fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 -- 172.21.15.77:0/88903202 shutdown_connections 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.864+0000 7f15906ac640 1 -- 172.21.15.77:0/88903202 wait complete. 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 Processor -- start 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 -- start start 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15881006e0 0x7f15881968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15881018d0 0x7f1588196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1588102290 0x7f158819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f158810c1f0 con 0x7f15881018d0 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f158810c070 con 0x7f15881006e0 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f15906ac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f158810c370 con 0x7f1588102290 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15881018d0 0x7f1588196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15881018d0 0x7f1588196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42380/0 (socket says 172.21.15.77:42380) 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 -- 172.21.15.77:0/172162645 learned_addr learned my addr 172.21.15.77:0/172162645 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158e448640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15881006e0 0x7f15881968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158ec49640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1588102290 0x7f158819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 -- 172.21.15.77:0/172162645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1588102290 msgr2=0x7f158819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1588102290 0x7f158819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 -- 172.21.15.77:0/172162645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15881006e0 msgr2=0x7f15881968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15881006e0 0x7f15881968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 -- 172.21.15.77:0/172162645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158819e510 con 0x7f15881018d0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.865+0000 7f158dc47640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15881018d0 0x7f1588196e30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f157800e9f0 tx=0x7f157800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.866+0000 7f157f7fe640 1 -- 172.21.15.77:0/172162645 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157800cd80 con 0x7f15881018d0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.866+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158819e800 con 0x7f15881018d0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.866+0000 7f157f7fe640 1 -- 172.21.15.77:0/172162645 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15780108e0 con 0x7f15881018d0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.866+0000 7f157f7fe640 1 -- 172.21.15.77:0/172162645 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578018920 con 0x7f15881018d0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.866+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158819ed50 con 0x7f15881018d0 2024-03-16T09:45:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.866+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1588107ad0 con 0x7f15881018d0 2024-03-16T09:45:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.868+0000 7f157f7fe640 1 -- 172.21.15.77:0/172162645 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1578010a90 con 0x7f15881018d0 2024-03-16T09:45:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.868+0000 7f157f7fe640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.868+0000 7f158e448640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.868+0000 7f157f7fe640 1 -- 172.21.15.77:0/172162645 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f15780cf160 con 0x7f15881018d0 2024-03-16T09:45:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.869+0000 7f158e448640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f15740a2800 0x7f15740a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1584005fd0 tx=0x7f158400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:46.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:46.872+0000 7f157f7fe640 1 -- 172.21.15.77:0/172162645 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f157809bbc0 con 0x7f15881018d0 2024-03-16T09:45:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.047+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f158802d340 con 0x7f15881018d0 2024-03-16T09:45:47.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.048+0000 7f157f7fe640 1 -- 172.21.15.77:0/172162645 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f157809fc60 con 0x7f15881018d0 2024-03-16T09:45:47.051 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f15740a2800 msgr2=0x7f15740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f15740a2800 0x7f15740a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1584005fd0 tx=0x7f158400b410 comp rx=0 tx=0).stop 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15881018d0 msgr2=0x7f1588196e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15881018d0 0x7f1588196e30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f157800e9f0 tx=0x7f157800eeb0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 shutdown_connections 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f15740a2800 0x7f15740a4cc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1588102290 0x7f158819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15881018d0 0x7f1588196e30 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 --2- 172.21.15.77:0/172162645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15881006e0 0x7f15881968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 >> 172.21.15.77:0/172162645 conn(0x7f15880fbc70 msgr2=0x7f1588104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 shutdown_connections 2024-03-16T09:45:47.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.050+0000 7f15906ac640 1 -- 172.21.15.77:0/172162645 wait complete. 2024-03-16T09:45:47.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:46 smithi077 ceph-mon[32171]: pgmap v462: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:47.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:46 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2382359326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:47.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:46 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2839677642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:47.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.164+0000 7faa65026640 1 -- 172.21.15.77:0/4044426100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 msgr2=0x7faa60101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.164+0000 7faa65026640 1 --2- 172.21.15.77:0/4044426100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60101d70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7faa48009900 tx=0x7faa4802bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 -- 172.21.15.77:0/4044426100 shutdown_connections 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 --2- 172.21.15.77:0/4044426100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa601022b0 0x7faa60108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 --2- 172.21.15.77:0/4044426100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60101d70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 --2- 172.21.15.77:0/4044426100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa60100700 0x7faa60100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 -- 172.21.15.77:0/4044426100 >> 172.21.15.77:0/4044426100 conn(0x7faa600fbc90 msgr2=0x7faa600fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 -- 172.21.15.77:0/4044426100 shutdown_connections 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 -- 172.21.15.77:0/4044426100 wait complete. 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 Processor -- start 2024-03-16T09:45:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.165+0000 7faa65026640 1 -- start start 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa65026640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa60100700 0x7faa60196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa65026640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa65026640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa601022b0 0x7faa6019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa65026640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faa6010c210 con 0x7faa601018f0 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa65026640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7faa6010c090 con 0x7faa60100700 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa65026640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa6010c390 con 0x7faa601022b0 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42402/0 (socket says 172.21.15.77:42402) 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 -- 172.21.15.77:0/1482618110 learned_addr learned my addr 172.21.15.77:0/1482618110 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5f59e640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa601022b0 0x7faa6019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5ed9d640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa60100700 0x7faa60196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 -- 172.21.15.77:0/1482618110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa601022b0 msgr2=0x7faa6019ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa601022b0 0x7faa6019ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 -- 172.21.15.77:0/1482618110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa60100700 msgr2=0x7faa60196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa60100700 0x7faa60196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 -- 172.21.15.77:0/1482618110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa6019e540 con 0x7faa601018f0 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5f59e640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa601022b0 0x7faa6019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa5e59c640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60196e60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7faa48006010 tx=0x7faa48005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa57fff640 1 -- 172.21.15.77:0/1482618110 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48043020 con 0x7faa601018f0 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa6019e7d0 con 0x7faa601018f0 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa57fff640 1 -- 172.21.15.77:0/1482618110 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa480043d0 con 0x7faa601018f0 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.166+0000 7faa57fff640 1 -- 172.21.15.77:0/1482618110 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48003b80 con 0x7faa601018f0 2024-03-16T09:45:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.167+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6019ed70 con 0x7faa601018f0 2024-03-16T09:45:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.167+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa60107a70 con 0x7faa601018f0 2024-03-16T09:45:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.168+0000 7faa57fff640 1 -- 172.21.15.77:0/1482618110 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faa48046020 con 0x7faa601018f0 2024-03-16T09:45:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.169+0000 7faa57fff640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa4c0a2800 0x7faa4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.169+0000 7faa5ed9d640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa4c0a2800 0x7faa4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.170+0000 7faa57fff640 1 -- 172.21.15.77:0/1482618110 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7faa480f12c0 con 0x7faa601018f0 2024-03-16T09:45:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.170+0000 7faa5ed9d640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa4c0a2800 0x7faa4c0a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7faa60101750 tx=0x7faa50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:47.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.173+0000 7faa57fff640 1 -- 172.21.15.77:0/1482618110 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa4804d020 con 0x7faa601018f0 2024-03-16T09:45:47.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:46 smithi129 ceph-mon[35841]: pgmap v462: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:47.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:46 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2382359326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:47.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:46 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2839677642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:47.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.349+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faa6002d1b0 con 0x7faa601018f0 2024-03-16T09:45:47.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.349+0000 7faa57fff640 1 -- 172.21.15.77:0/1482618110 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7faa480be310 con 0x7faa601018f0 2024-03-16T09:45:47.352 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:45:47.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa4c0a2800 msgr2=0x7faa4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa4c0a2800 0x7faa4c0a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7faa60101750 tx=0x7faa50005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 msgr2=0x7faa60196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60196e60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7faa48006010 tx=0x7faa48005e40 comp rx=0 tx=0).stop 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 shutdown_connections 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7faa4c0a2800 0x7faa4c0a4cc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa601022b0 0x7faa6019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa601018f0 0x7faa60196e60 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 --2- 172.21.15.77:0/1482618110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa60100700 0x7faa60196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.351+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 >> 172.21.15.77:0/1482618110 conn(0x7faa600fbc90 msgr2=0x7faa60104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.352+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 shutdown_connections 2024-03-16T09:45:47.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.352+0000 7faa65026640 1 -- 172.21.15.77:0/1482618110 wait complete. 2024-03-16T09:45:47.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T09:45:47.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:46 smithi119 ceph-mon[35632]: pgmap v462: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T09:45:47.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:46 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2382359326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:47.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:46 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2839677642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1633127028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf04105390 msgr2=0x7fcf04105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1633127028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf04105390 0x7fcf04105810 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009ca0 tx=0x7fcefc02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1633127028 shutdown_connections 2024-03-16T09:45:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1633127028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf04105390 0x7fcf04105810 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1633127028 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf041029e0 0x7fcf04104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1633127028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf04100070 0x7fcf041024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1633127028 >> 172.21.15.77:0/1633127028 conn(0x7fcf040fbc90 msgr2=0x7fcf040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1633127028 shutdown_connections 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.465+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1633127028 wait complete. 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 Processor -- start 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 -- start start 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf04100070 0x7fcf0419dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf041029e0 0x7fcf0419e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf04105390 0x7fcf041a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf04113920 con 0x7fcf041029e0 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041137a0 con 0x7fcf04100070 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0b2ff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcf04113aa0 con 0x7fcf04105390 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf041029e0 0x7fcf0419e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf041029e0 0x7fcf0419e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42410/0 (socket says 172.21.15.77:42410) 2024-03-16T09:45:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 -- 172.21.15.77:0/1810924037 learned_addr learned my addr 172.21.15.77:0/1810924037 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0989c640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf04105390 0x7fcf041a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0909b640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf04100070 0x7fcf0419dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 -- 172.21.15.77:0/1810924037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf04105390 msgr2=0x7fcf041a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf04105390 0x7fcf041a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 -- 172.21.15.77:0/1810924037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf04100070 msgr2=0x7fcf0419dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf04100070 0x7fcf0419dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.466+0000 7fcf0889a640 1 -- 172.21.15.77:0/1810924037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf041a5bf0 con 0x7fcf041029e0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.467+0000 7fcf0889a640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf041029e0 0x7fcf0419e510 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9f0 tx=0x7fcef400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.467+0000 7fcefa7fc640 1 -- 172.21.15.77:0/1810924037 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef400cd80 con 0x7fcf041029e0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.467+0000 7fcefa7fc640 1 -- 172.21.15.77:0/1810924037 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcef40108e0 con 0x7fcf041029e0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.467+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf041a5ee0 con 0x7fcf041029e0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.467+0000 7fcefa7fc640 1 -- 172.21.15.77:0/1810924037 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4018920 con 0x7fcf041029e0 2024-03-16T09:45:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.467+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf041a63a0 con 0x7fcf041029e0 2024-03-16T09:45:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.468+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf040fd780 con 0x7fcf041029e0 2024-03-16T09:45:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.469+0000 7fcefa7fc640 1 -- 172.21.15.77:0/1810924037 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcef4010a90 con 0x7fcf041029e0 2024-03-16T09:45:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.470+0000 7fcefa7fc640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcef00a2800 0x7fcef00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.470+0000 7fcf0909b640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcef00a2800 0x7fcef00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.470+0000 7fcefa7fc640 1 -- 172.21.15.77:0/1810924037 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcef40ced60 con 0x7fcf041029e0 2024-03-16T09:45:47.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.470+0000 7fcf0909b640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcef00a2800 0x7fcef00a4cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009bd0 tx=0x7fcf00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:47.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.474+0000 7fcefa7fc640 1 -- 172.21.15.77:0/1810924037 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcef409b7c0 con 0x7fcf041029e0 2024-03-16T09:45:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.661+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcf0402d5b0 con 0x7fcf041029e0 2024-03-16T09:45:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.661+0000 7fcefa7fc640 1 -- 172.21.15.77:0/1810924037 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+590 (secure 0 0 0) 0x7fcef409f860 con 0x7fcf041029e0 2024-03-16T09:45:47.664 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/1569662437 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:47.664 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/3524537702 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6801/3741578273 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/2291840305 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/2778231529 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6800/262452209 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6801/262452209 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/2049888475 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/3919406383 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6800/3741578273 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:47.665 INFO:teuthology.orchestra.run.smithi077.stderr:listed 10 entries 2024-03-16T09:45:47.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcef00a2800 msgr2=0x7fcef00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcef00a2800 0x7fcef00a4cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009bd0 tx=0x7fcf00005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:47.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf041029e0 msgr2=0x7fcf0419e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf041029e0 0x7fcf0419e510 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9f0 tx=0x7fcef400eeb0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 shutdown_connections 2024-03-16T09:45:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fcef00a2800 0x7fcef00a4cc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf04105390 0x7fcf041a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf041029e0 0x7fcf0419e510 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 --2- 172.21.15.77:0/1810924037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf04100070 0x7fcf0419dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 >> 172.21.15.77:0/1810924037 conn(0x7fcf040fbc90 msgr2=0x7fcf04100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 shutdown_connections 2024-03-16T09:45:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.663+0000 7fcf0b2ff640 1 -- 172.21.15.77:0/1810924037 wait complete. 2024-03-16T09:45:47.677 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:45:47.677 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T09:45:47.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:45:47.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 -- 172.21.15.77:0/2498976235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018101970 msgr2=0x7f5018101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 --2- 172.21.15.77:0/2498976235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018101970 0x7f5018101df0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5008009900 tx=0x7f500802bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 -- 172.21.15.77:0/2498976235 shutdown_connections 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 --2- 172.21.15.77:0/2498976235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5018102330 0x7f5018108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 --2- 172.21.15.77:0/2498976235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018101970 0x7f5018101df0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 --2- 172.21.15.77:0/2498976235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5018100780 0x7f5018100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 -- 172.21.15.77:0/2498976235 >> 172.21.15.77:0/2498976235 conn(0x7f50180fbd10 msgr2=0x7f50180fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 -- 172.21.15.77:0/2498976235 shutdown_connections 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 -- 172.21.15.77:0/2498976235 wait complete. 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 Processor -- start 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.833+0000 7f501f5e8640 1 -- start start 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5018100780 0x7f501806a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5018101970 0x7f501819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018102330 0x7f501819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f501810c220 con 0x7f5018102330 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f501810c0a0 con 0x7f5018100780 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f501810c3a0 con 0x7f5018101970 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018102330 0x7f501819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018102330 0x7f501819eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42440/0 (socket says 172.21.15.77:42440) 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 -- 172.21.15.77:0/1966383811 learned_addr learned my addr 172.21.15.77:0/1966383811 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501d384640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5018100780 0x7f501806a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501cb83640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5018101970 0x7f501819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 -- 172.21.15.77:0/1966383811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5018101970 msgr2=0x7f501819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5018101970 0x7f501819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 -- 172.21.15.77:0/1966383811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5018100780 msgr2=0x7f501806a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5018100780 0x7f501806a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 -- 172.21.15.77:0/1966383811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501819f660 con 0x7f5018102330 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501db85640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018102330 0x7f501819eda0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f501400e970 tx=0x7f501400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f500e7fc640 1 -- 172.21.15.77:0/1966383811 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501400ccc0 con 0x7f5018102330 2024-03-16T09:45:47.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f500e7fc640 1 -- 172.21.15.77:0/1966383811 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f501400ce60 con 0x7f5018102330 2024-03-16T09:45:47.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f500e7fc640 1 -- 172.21.15.77:0/1966383811 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5014010430 con 0x7f5018102330 2024-03-16T09:45:47.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501819f950 con 0x7f5018102330 2024-03-16T09:45:47.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.834+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f501819fe20 con 0x7f5018102330 2024-03-16T09:45:47.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.835+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5018107bc0 con 0x7f5018102330 2024-03-16T09:45:47.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.836+0000 7f500e7fc640 1 -- 172.21.15.77:0/1966383811 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5014004ff0 con 0x7f5018102330 2024-03-16T09:45:47.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.837+0000 7f500e7fc640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50040a2800 0x7f50040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:47.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.837+0000 7f501d384640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50040a2800 0x7f50040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:47.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.837+0000 7f500e7fc640 1 -- 172.21.15.77:0/1966383811 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f50140cee60 con 0x7f5018102330 2024-03-16T09:45:47.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.837+0000 7f501d384640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50040a2800 0x7f50040a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f50100099a0 tx=0x7f5010009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:47.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:47.842+0000 7f500e7fc640 1 -- 172.21.15.77:0/1966383811 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f501409bf10 con 0x7f5018102330 2024-03-16T09:45:48.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.022+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f501802d1b0 con 0x7f5018102330 2024-03-16T09:45:48.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.024+0000 7f500e7fc640 1 -- 172.21.15.77:0/1966383811 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f501409b810 con 0x7f5018102330 2024-03-16T09:45:48.027 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:48.027 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:45:48.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50040a2800 msgr2=0x7f50040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50040a2800 0x7f50040a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f50100099a0 tx=0x7f5010009500 comp rx=0 tx=0).stop 2024-03-16T09:45:48.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018102330 msgr2=0x7f501819eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018102330 0x7f501819eda0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f501400e970 tx=0x7f501400ed70 comp rx=0 tx=0).stop 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 shutdown_connections 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f50040a2800 0x7f50040a4cc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5018102330 0x7f501819eda0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5018101970 0x7f501819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 --2- 172.21.15.77:0/1966383811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5018100780 0x7f501806a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 >> 172.21.15.77:0/1966383811 conn(0x7f50180fbd10 msgr2=0x7f5018104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 shutdown_connections 2024-03-16T09:45:48.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.026+0000 7f501f5e8640 1 -- 172.21.15.77:0/1966383811 wait complete. 2024-03-16T09:45:48.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T09:35:24.036758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T09:39:54.014341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T09:39:54.733977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T09:45:48.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 -- 172.21.15.77:0/2193547098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 msgr2=0x7ff864108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/2193547098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864108b30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009ca0 tx=0x7ff85c02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 -- 172.21.15.77:0/2193547098 shutdown_connections 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/2193547098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864108b30 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/2193547098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8641018e0 0x7ff864101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/2193547098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8641006f0 0x7ff864100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 -- 172.21.15.77:0/2193547098 >> 172.21.15.77:0/2193547098 conn(0x7ff8640fbc80 msgr2=0x7ff8640fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 -- 172.21.15.77:0/2193547098 shutdown_connections 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 -- 172.21.15.77:0/2193547098 wait complete. 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 Processor -- start 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.144+0000 7ff86b5c6640 1 -- start start 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8641006f0 0x7ff86410b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8641018e0 0x7ff8641096b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff86410d340 con 0x7ff8641022a0 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff86410d1c0 con 0x7ff8641006f0 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff86410d4c0 con 0x7ff8641018e0 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864109bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42458/0 (socket says 172.21.15.77:42458) 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 -- 172.21.15.77:0/445105620 learned_addr learned my addr 172.21.15.77:0/445105620 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:48.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869362640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8641006f0 0x7ff86410b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff868b61640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8641018e0 0x7ff8641096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 -- 172.21.15.77:0/445105620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8641018e0 msgr2=0x7ff8641096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8641018e0 0x7ff8641096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 -- 172.21.15.77:0/445105620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8641006f0 msgr2=0x7ff86410b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8641006f0 0x7ff86410b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 -- 172.21.15.77:0/445105620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86410a480 con 0x7ff8641022a0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff869b63640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864109bf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff85c02c700 tx=0x7ff85c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff85a7fc640 1 -- 172.21.15.77:0/445105620 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c041020 con 0x7ff8641022a0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff85a7fc640 1 -- 172.21.15.77:0/445105620 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff85c003e20 con 0x7ff8641022a0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8641a1a90 con 0x7ff8641022a0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff85a7fc640 1 -- 172.21.15.77:0/445105620 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c039040 con 0x7ff8641022a0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.145+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8641a1fe0 con 0x7ff8641022a0 2024-03-16T09:45:48.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.146+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff864068a20 con 0x7ff8641022a0 2024-03-16T09:45:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.147+0000 7ff85a7fc640 1 -- 172.21.15.77:0/445105620 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff85c044020 con 0x7ff8641022a0 2024-03-16T09:45:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.148+0000 7ff85a7fc640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff8500a2800 0x7ff8500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.148+0000 7ff869362640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff8500a2800 0x7ff8500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.148+0000 7ff85a7fc640 1 -- 172.21.15.77:0/445105620 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff85c0f0c50 con 0x7ff8641022a0 2024-03-16T09:45:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.148+0000 7ff869362640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff8500a2800 0x7ff8500a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff864101740 tx=0x7ff860005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.152+0000 7ff85a7fc640 1 -- 172.21.15.77:0/445105620 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff85c0f6020 con 0x7ff8641022a0 2024-03-16T09:45:48.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:47 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/172162645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:48.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:47 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1482618110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:48.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:47 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1810924037' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T09:45:48.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:47 smithi077 ceph-mon[32171]: pgmap v463: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:48.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:47 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/172162645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:48.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:47 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1482618110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:48.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:47 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1810924037' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T09:45:48.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:47 smithi129 ceph-mon[35841]: pgmap v463: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:48.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.333+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff8640623e0 con 0x7ff8641022a0 2024-03-16T09:45:48.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.333+0000 7ff85a7fc640 1 -- 172.21.15.77:0/445105620 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7ff85c04b020 con 0x7ff8641022a0 2024-03-16T09:45:48.336 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T09:45:48.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff8500a2800 msgr2=0x7ff8500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff8500a2800 0x7ff8500a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff864101740 tx=0x7ff860005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:48.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 msgr2=0x7ff864109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864109bf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff85c02c700 tx=0x7ff85c0328a0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 shutdown_connections 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7ff8500a2800 0x7ff8500a4cc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8641022a0 0x7ff864109bf0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8641018e0 0x7ff8641096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 --2- 172.21.15.77:0/445105620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8641006f0 0x7ff86410b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 >> 172.21.15.77:0/445105620 conn(0x7ff8640fbc80 msgr2=0x7ff864104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 shutdown_connections 2024-03-16T09:45:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.335+0000 7ff86b5c6640 1 -- 172.21.15.77:0/445105620 wait complete. 2024-03-16T09:45:48.349 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T09:45:48.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:47 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/172162645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:48.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:47 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1482618110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:48.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:47 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1810924037' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T09:45:48.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:47 smithi119 ceph-mon[35632]: pgmap v463: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T09:45:48.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- 172.21.15.77:0/3159643252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c1018d0 msgr2=0x7f682c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 --2- 172.21.15.77:0/3159643252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c1018d0 0x7f682c101d50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6820009900 tx=0x7f682002bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:48.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- 172.21.15.77:0/3159643252 shutdown_connections 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 --2- 172.21.15.77:0/3159643252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f682c102290 0x7f682c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 --2- 172.21.15.77:0/3159643252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c1018d0 0x7f682c101d50 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 --2- 172.21.15.77:0/3159643252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f682c1006e0 0x7f682c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- 172.21.15.77:0/3159643252 >> 172.21.15.77:0/3159643252 conn(0x7f682c0fbc70 msgr2=0x7f682c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- 172.21.15.77:0/3159643252 shutdown_connections 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- 172.21.15.77:0/3159643252 wait complete. 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 Processor -- start 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- start start 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f682c1006e0 0x7f682c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f682c1018d0 0x7f682c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c102290 0x7f682c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f682c10d330 con 0x7f682c102290 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.452+0000 7f68317d4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f682c10d1b0 con 0x7f682c1018d0 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f68317d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f682c10d4b0 con 0x7f682c1006e0 2024-03-16T09:45:48.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c102290 0x7f682c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682a7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f682c1018d0 0x7f682c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f682c1006e0 0x7f682c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682a7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f682c1018d0 0x7f682c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.77:46562/0 (socket says 172.21.15.77:46562) 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682a7fc640 1 -- 172.21.15.77:0/342706852 learned_addr learned my addr 172.21.15.77:0/342706852 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682b7fe640 1 -- 172.21.15.77:0/342706852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f682c1006e0 msgr2=0x7f682c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682b7fe640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f682c1006e0 0x7f682c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682b7fe640 1 -- 172.21.15.77:0/342706852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f682c1018d0 msgr2=0x7f682c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682b7fe640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f682c1018d0 0x7f682c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682b7fe640 1 -- 172.21.15.77:0/342706852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c19f650 con 0x7f682c102290 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f682b7fe640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c102290 0x7f682c19efe0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6824009fd0 tx=0x7f682400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f6813fff640 1 -- 172.21.15.77:0/342706852 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682400ce20 con 0x7f682c102290 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682c19f940 con 0x7f682c102290 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f6813fff640 1 -- 172.21.15.77:0/342706852 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6824005020 con 0x7f682c102290 2024-03-16T09:45:48.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f6813fff640 1 -- 172.21.15.77:0/342706852 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824010690 con 0x7f682c102290 2024-03-16T09:45:48.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.453+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f682c19fee0 con 0x7f682c102290 2024-03-16T09:45:48.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.454+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682c107b20 con 0x7f682c102290 2024-03-16T09:45:48.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.455+0000 7f6813fff640 1 -- 172.21.15.77:0/342706852 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f68240051d0 con 0x7f682c102290 2024-03-16T09:45:48.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.456+0000 7f6813fff640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f68140a2800 0x7f68140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.456+0000 7f682affd640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f68140a2800 0x7f68140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.456+0000 7f6813fff640 1 -- 172.21.15.77:0/342706852 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f68240cf0a0 con 0x7f682c102290 2024-03-16T09:45:48.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.456+0000 7f682affd640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f68140a2800 0x7f68140a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f682c101730 tx=0x7f681c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:48.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.459+0000 7f6813fff640 1 -- 172.21.15.77:0/342706852 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f682409c150 con 0x7f682c102290 2024-03-16T09:45:48.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.635+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f682c02d1b0 con 0x7f682c102290 2024-03-16T09:45:48.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.636+0000 7f6813fff640 1 -- 172.21.15.77:0/342706852 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f682409ba50 con 0x7f682c102290 2024-03-16T09:45:48.639 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T09:45:48.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f68140a2800 msgr2=0x7f68140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f68140a2800 0x7f68140a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f682c101730 tx=0x7f681c005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:48.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c102290 msgr2=0x7f682c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c102290 0x7f682c19efe0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6824009fd0 tx=0x7f682400ef10 comp rx=0 tx=0).stop 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 shutdown_connections 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f68140a2800 0x7f68140a4cc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f682c102290 0x7f682c19efe0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f682c1018d0 0x7f682c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 --2- 172.21.15.77:0/342706852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f682c1006e0 0x7f682c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 >> 172.21.15.77:0/342706852 conn(0x7f682c0fbc70 msgr2=0x7f682c104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 shutdown_connections 2024-03-16T09:45:48.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.638+0000 7f68317d4640 1 -- 172.21.15.77:0/342706852 wait complete. 2024-03-16T09:45:48.652 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T09:45:48.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 -- 172.21.15.77:0/2346764210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63641018f0 msgr2=0x7f6364101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 --2- 172.21.15.77:0/2346764210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63641018f0 0x7f6364101d70 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6350009900 tx=0x7f635002bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:48.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 -- 172.21.15.77:0/2346764210 shutdown_connections 2024-03-16T09:45:48.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 --2- 172.21.15.77:0/2346764210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63641022b0 0x7f6364108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 --2- 172.21.15.77:0/2346764210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63641018f0 0x7f6364101d70 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 --2- 172.21.15.77:0/2346764210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6364100700 0x7f6364100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 -- 172.21.15.77:0/2346764210 >> 172.21.15.77:0/2346764210 conn(0x7f63640fbc90 msgr2=0x7f63640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 -- 172.21.15.77:0/2346764210 shutdown_connections 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.754+0000 7f636af60640 1 -- 172.21.15.77:0/2346764210 wait complete. 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 Processor -- start 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 -- start start 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6364100700 0x7f6364197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63641018f0 0x7f6364197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63641022b0 0x7f636419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f636410c210 con 0x7f6364100700 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f636410c090 con 0x7f63641022b0 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f636af60640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f636410c390 con 0x7f63641018f0 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6364100700 0x7f6364197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6364100700 0x7f6364197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42504/0 (socket says 172.21.15.77:42504) 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 -- 172.21.15.77:0/1553887735 learned_addr learned my addr 172.21.15.77:0/1553887735 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6363fff640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63641018f0 0x7f6364197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f63694fd640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63641022b0 0x7f636419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 -- 172.21.15.77:0/1553887735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63641018f0 msgr2=0x7f6364197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63641018f0 0x7f6364197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 -- 172.21.15.77:0/1553887735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63641022b0 msgr2=0x7f636419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63641022b0 0x7f636419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 -- 172.21.15.77:0/1553887735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636419f680 con 0x7f6364100700 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.755+0000 7f6368cfc640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6364100700 0x7f6364197a60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f635800e9c0 tx=0x7f635800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.756+0000 7f6361ffb640 1 -- 172.21.15.77:0/1553887735 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635800cd80 con 0x7f6364100700 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.756+0000 7f6361ffb640 1 -- 172.21.15.77:0/1553887735 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63580035f0 con 0x7f6364100700 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.756+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f636419f970 con 0x7f6364100700 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.756+0000 7f6361ffb640 1 -- 172.21.15.77:0/1553887735 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358010690 con 0x7f6364100700 2024-03-16T09:45:48.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.756+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f636419fe90 con 0x7f6364100700 2024-03-16T09:45:48.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.757+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6364068a20 con 0x7f6364100700 2024-03-16T09:45:48.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.758+0000 7f6361ffb640 1 -- 172.21.15.77:0/1553887735 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f63580037a0 con 0x7f6364100700 2024-03-16T09:45:48.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.758+0000 7f6361ffb640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f63540a2800 0x7f63540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:48.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.758+0000 7f6363fff640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f63540a2800 0x7f63540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:48.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.758+0000 7f6361ffb640 1 -- 172.21.15.77:0/1553887735 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f63580cede0 con 0x7f6364100700 2024-03-16T09:45:48.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.759+0000 7f6363fff640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f63540a2800 0x7f63540a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6350009230 tx=0x7f6350005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:48.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.762+0000 7f6361ffb640 1 -- 172.21.15.77:0/1553887735 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f635809be90 con 0x7f6364100700 2024-03-16T09:45:48.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.941+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f63641a0800 con 0x7f6364100700 2024-03-16T09:45:48.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.942+0000 7f6361ffb640 1 -- 172.21.15.77:0/1553887735 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f635809b790 con 0x7f6364100700 2024-03-16T09:45:48.945 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f63540a2800 msgr2=0x7f63540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f63540a2800 0x7f63540a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6350009230 tx=0x7f6350005fd0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6364100700 msgr2=0x7f6364197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6364100700 0x7f6364197a60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f635800e9c0 tx=0x7f635800ee80 comp rx=0 tx=0).stop 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 shutdown_connections 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f63540a2800 0x7f63540a4cc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63641022b0 0x7f636419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63641018f0 0x7f6364197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 --2- 172.21.15.77:0/1553887735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6364100700 0x7f6364197a60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 >> 172.21.15.77:0/1553887735 conn(0x7f63640fbc90 msgr2=0x7f6364069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:48.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 shutdown_connections 2024-03-16T09:45:48.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:48.944+0000 7f636af60640 1 -- 172.21.15.77:0/1553887735 wait complete. 2024-03-16T09:45:48.958 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T09:45:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 -- 172.21.15.77:0/2702609014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 msgr2=0x7f47d8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 --2- 172.21.15.77:0/2702609014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d8100b20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009900 tx=0x7f47c802bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 -- 172.21.15.77:0/2702609014 shutdown_connections 2024-03-16T09:45:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 --2- 172.21.15.77:0/2702609014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d81022b0 0x7f47d8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 --2- 172.21.15.77:0/2702609014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47d81018f0 0x7f47d8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 --2- 172.21.15.77:0/2702609014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d8100b20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 -- 172.21.15.77:0/2702609014 >> 172.21.15.77:0/2702609014 conn(0x7f47d80fbc90 msgr2=0x7f47d80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 -- 172.21.15.77:0/2702609014 shutdown_connections 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.058+0000 7f47df8dc640 1 -- 172.21.15.77:0/2702609014 wait complete. 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 Processor -- start 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 -- start start 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d806a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d81018f0 0x7f47d819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47d81022b0 0x7f47d819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f47d810c230 con 0x7f47d8100700 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f47d810c0b0 con 0x7f47d81022b0 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47df8dc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47d810c3b0 con 0x7f47d81018f0 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d806a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:49.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d806a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42522/0 (socket says 172.21.15.77:42522) 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 -- 172.21.15.77:0/1553505133 learned_addr learned my addr 172.21.15.77:0/1553505133 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dde79640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47d81022b0 0x7f47d819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dce77640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d81018f0 0x7f47d819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 -- 172.21.15.77:0/1553505133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d81018f0 msgr2=0x7f47d819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d81018f0 0x7f47d819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 -- 172.21.15.77:0/1553505133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47d81022b0 msgr2=0x7f47d819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47d81022b0 0x7f47d819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 -- 172.21.15.77:0/1553505133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d819f670 con 0x7f47d8100700 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47dd678640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d806a980 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f47c8000c00 tx=0x7f47c8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47ce7fc640 1 -- 172.21.15.77:0/1553505133 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8043020 con 0x7f47d8100700 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.059+0000 7f47ce7fc640 1 -- 172.21.15.77:0/1553505133 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c8004350 con 0x7f47d8100700 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.060+0000 7f47ce7fc640 1 -- 172.21.15.77:0/1553505133 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003b80 con 0x7f47d8100700 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.060+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d819f900 con 0x7f47d8100700 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.060+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d819fe50 con 0x7f47d8100700 2024-03-16T09:45:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.061+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d8107b10 con 0x7f47d8100700 2024-03-16T09:45:49.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.061+0000 7f47ce7fc640 1 -- 172.21.15.77:0/1553505133 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f47c8046020 con 0x7f47d8100700 2024-03-16T09:45:49.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.062+0000 7f47ce7fc640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47c40a2800 0x7f47c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:49.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.062+0000 7f47dce77640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47c40a2800 0x7f47c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:49.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.062+0000 7f47ce7fc640 1 -- 172.21.15.77:0/1553505133 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f47c80f0b70 con 0x7f47d8100700 2024-03-16T09:45:49.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.062+0000 7f47dce77640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47c40a2800 0x7f47c40a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f47d806a6f0 tx=0x7f47d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:49.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.066+0000 7f47ce7fc640 1 -- 172.21.15.77:0/1553505133 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f47c804d020 con 0x7f47d8100700 2024-03-16T09:45:49.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.255+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f47d80623e0 con 0x7f47d8100700 2024-03-16T09:45:49.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.255+0000 7f47ce7fc640 1 -- 172.21.15.77:0/1553505133 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+590 (secure 0 0 0) 0x7f47c80bdc20 con 0x7f47d8100700 2024-03-16T09:45:49.258 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/1569662437 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:49.258 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/3524537702 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:49.258 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6801/3741578273 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:49.258 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/2291840305 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/2778231529 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6800/262452209 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6801/262452209 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/2049888475 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/3919406383 2024-03-17T09:32:29.772291+0000 2024-03-16T09:45:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6800/3741578273 2024-03-17T09:32:57.693064+0000 2024-03-16T09:45:49.259 INFO:teuthology.orchestra.run.smithi077.stderr:listed 10 entries 2024-03-16T09:45:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:48 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1966383811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:49.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:48 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/445105620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:48 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/342706852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:48 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1553887735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47c40a2800 msgr2=0x7f47c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:49.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47c40a2800 0x7f47c40a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f47d806a6f0 tx=0x7f47d4005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 msgr2=0x7f47d806a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d806a980 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f47c8000c00 tx=0x7f47c8005dc0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 shutdown_connections 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f47c40a2800 0x7f47c40a4cc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47d81022b0 0x7f47d819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47d81018f0 0x7f47d819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 --2- 172.21.15.77:0/1553505133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47d8100700 0x7f47d806a980 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 >> 172.21.15.77:0/1553505133 conn(0x7f47d80fbc90 msgr2=0x7f47d8104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 shutdown_connections 2024-03-16T09:45:49.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:49.257+0000 7f47df8dc640 1 -- 172.21.15.77:0/1553505133 wait complete. 2024-03-16T09:45:49.271 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi077.front.sepia.ceph.com... 2024-03-16T09:45:49.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:45:49.271 DEBUG:teuthology.orchestra.run.smithi077:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-16T09:45:49.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T09:45:49.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:48 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1966383811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:48 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/445105620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:48 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/342706852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:48 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1553887735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi077.stderr:ceph-fuse[84414]: fuse finished with error 0 and tester_r 0 2024-03-16T09:45:49.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:48 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1966383811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:49.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:48 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/445105620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:48 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/342706852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T09:45:49.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:48 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1553887735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T09:45:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:49 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1553505133' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T09:45:50.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:49 smithi077 ceph-mon[32171]: pgmap v464: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:50.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:49 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1553505133' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T09:45:50.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:49 smithi129 ceph-mon[35841]: pgmap v464: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:50.439 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:49 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1553505133' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T09:45:50.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:49 smithi119 ceph-mon[35632]: pgmap v464: 97 pgs: 97 active+clean; 979 KiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T09:45:53.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:52 smithi119 ceph-mon[35632]: pgmap v465: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T09:45:53.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:52 smithi077 ceph-mon[32171]: pgmap v465: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T09:45:53.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:52 smithi129 ceph-mon[35841]: pgmap v465: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T09:45:55.188 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:54 smithi077 ceph-mon[32171]: pgmap v466: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T09:45:55.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:54 smithi119 ceph-mon[35632]: pgmap v466: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T09:45:55.300 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:54 smithi129 ceph-mon[35841]: pgmap v466: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T09:45:55.401 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:45:55.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:45:55.401 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-16T09:45:55.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:45:55.428 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-16T09:45:55.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-16T09:45:55.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:45:55.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T09:45:55.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T09:45:55.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-03-16T09:45:55.653 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-16T09:45:55.664 INFO:tasks.cephadm:Teardown begin 2024-03-16T09:45:55.664 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-16T09:45:55.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:45:55.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 -- 172.21.15.77:0/2652368135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c1023c0 msgr2=0x7f344c1047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:55.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 --2- 172.21.15.77:0/2652368135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c1023c0 0x7f344c1047b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f343c009900 tx=0x7f343c034440 comp rx=0 tx=0).stop 2024-03-16T09:45:55.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 -- 172.21.15.77:0/2652368135 shutdown_connections 2024-03-16T09:45:55.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 --2- 172.21.15.77:0/2652368135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f344c107620 0x7f344c109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 --2- 172.21.15.77:0/2652368135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f344c104cf0 0x7f344c1070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 --2- 172.21.15.77:0/2652368135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c1023c0 0x7f344c1047b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 -- 172.21.15.77:0/2652368135 >> 172.21.15.77:0/2652368135 conn(0x7f344c0fbd10 msgr2=0x7f344c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 -- 172.21.15.77:0/2652368135 shutdown_connections 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.763+0000 7f345424b640 1 -- 172.21.15.77:0/2652368135 wait complete. 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 Processor -- start 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 -- start start 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f344c1023c0 0x7f344c195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c104cf0 0x7f344c195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f344c107620 0x7f344c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f344c10b150 con 0x7f344c104cf0 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f344c10afd0 con 0x7f344c1023c0 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f345424b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f344c10b2d0 con 0x7f344c107620 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c104cf0 0x7f344c195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c104cf0 0x7f344c195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56112/0 (socket says 172.21.15.77:56112) 2024-03-16T09:45:55.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 -- 172.21.15.77:0/2060936674 learned_addr learned my addr 172.21.15.77:0/2060936674 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f3451fe7640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f344c1023c0 0x7f344c195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34527e8640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f344c107620 0x7f344c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 -- 172.21.15.77:0/2060936674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f344c107620 msgr2=0x7f344c19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f344c107620 0x7f344c19ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 -- 172.21.15.77:0/2060936674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f344c1023c0 msgr2=0x7f344c195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f344c1023c0 0x7f344c195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 -- 172.21.15.77:0/2060936674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c19d480 con 0x7f344c104cf0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.764+0000 7f34517e6640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c104cf0 0x7f344c195da0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3448009a10 tx=0x7f3448009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.765+0000 7f3442ffd640 1 -- 172.21.15.77:0/2060936674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344800cc00 con 0x7f344c104cf0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.765+0000 7f3442ffd640 1 -- 172.21.15.77:0/2060936674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f344800cda0 con 0x7f344c104cf0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.765+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c19d770 con 0x7f344c104cf0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.765+0000 7f3442ffd640 1 -- 172.21.15.77:0/2060936674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3448017400 con 0x7f344c104cf0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.765+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c19dcc0 con 0x7f344c104cf0 2024-03-16T09:45:55.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.766+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344c0fd800 con 0x7f344c104cf0 2024-03-16T09:45:55.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.767+0000 7f3442ffd640 1 -- 172.21.15.77:0/2060936674 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3448004fe0 con 0x7f344c104cf0 2024-03-16T09:45:55.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.767+0000 7f3442ffd640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34380a2880 0x7f34380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:55.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.767+0000 7f3451fe7640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34380a2880 0x7f34380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:55.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.767+0000 7f3442ffd640 1 -- 172.21.15.77:0/2060936674 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f34480cf0c0 con 0x7f344c104cf0 2024-03-16T09:45:55.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.768+0000 7f3451fe7640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34380a2880 0x7f34380a4d40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f343c006010 tx=0x7f343c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:55.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.771+0000 7f3442ffd640 1 -- 172.21.15.77:0/2060936674 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f344809bb20 con 0x7f344c104cf0 2024-03-16T09:45:55.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.983+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f344c02d400 con 0x7f344c104cf0 2024-03-16T09:45:55.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.983+0000 7f3442ffd640 1 -- 172.21.15.77:0/2060936674 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f344809fbc0 con 0x7f344c104cf0 2024-03-16T09:45:55.986 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:55.986 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:45:55.987 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:45:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34380a2880 msgr2=0x7f34380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34380a2880 0x7f34380a4d40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f343c006010 tx=0x7f343c000bc0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c104cf0 msgr2=0x7f344c195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c104cf0 0x7f344c195da0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3448009a10 tx=0x7f3448009ed0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 shutdown_connections 2024-03-16T09:45:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f34380a2880 0x7f34380a4d40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f344c107620 0x7f344c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f344c104cf0 0x7f344c195da0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 --2- 172.21.15.77:0/2060936674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f344c1023c0 0x7f344c195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 >> 172.21.15.77:0/2060936674 conn(0x7f344c0fbd10 msgr2=0x7f344c102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 shutdown_connections 2024-03-16T09:45:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:55.985+0000 7f345424b640 1 -- 172.21.15.77:0/2060936674 wait complete. 2024-03-16T09:45:56.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T09:45:56.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3409556348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7381033e0 msgr2=0x7fc738109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3409556348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7381033e0 0x7fc738109c70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc730009900 tx=0x7fc73002bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:56.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3409556348 shutdown_connections 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3409556348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7381033e0 0x7fc738109c70 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3409556348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc738102a20 0x7fc738102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3409556348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc738101830 0x7fc738101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3409556348 >> 172.21.15.77:0/3409556348 conn(0x7fc7380fcd80 msgr2=0x7fc7380ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3409556348 shutdown_connections 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.098+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3409556348 wait complete. 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 Processor -- start 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 -- start start 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc738101830 0x7fc7381979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc738102a20 0x7fc738197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7381033e0 0x7fc73819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc73810d340 con 0x7fc738102a20 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc73810d1c0 con 0x7fc738101830 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc73d2ae640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc73810d4c0 con 0x7fc7381033e0 2024-03-16T09:45:56.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc7367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc738102a20 0x7fc738197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc7367fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc738102a20 0x7fc738197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56136/0 (socket says 172.21.15.77:56136) 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc7367fc640 1 -- 172.21.15.77:0/3448525602 learned_addr learned my addr 172.21.15.77:0/3448525602 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc736ffd640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc738101830 0x7fc7381979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.099+0000 7fc7377fe640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7381033e0 0x7fc73819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc7367fc640 1 -- 172.21.15.77:0/3448525602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7381033e0 msgr2=0x7fc73819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc7367fc640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7381033e0 0x7fc73819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc7367fc640 1 -- 172.21.15.77:0/3448525602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc738101830 msgr2=0x7fc7381979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc7367fc640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc738101830 0x7fc7381979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc7367fc640 1 -- 172.21.15.77:0/3448525602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73819f600 con 0x7fc738102a20 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc7367fc640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc738102a20 0x7fc738197f20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00d950 tx=0x7fc72c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc71ffff640 1 -- 172.21.15.77:0/3448525602 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c017070 con 0x7fc738102a20 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73819f8f0 con 0x7fc738102a20 2024-03-16T09:45:56.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc71ffff640 1 -- 172.21.15.77:0/3448525602 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc72c009c30 con 0x7fc738102a20 2024-03-16T09:45:56.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc71ffff640 1 -- 172.21.15.77:0/3448525602 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00f620 con 0x7fc738102a20 2024-03-16T09:45:56.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.100+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73819fe40 con 0x7fc738102a20 2024-03-16T09:45:56.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.101+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc738108bd0 con 0x7fc738102a20 2024-03-16T09:45:56.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.102+0000 7fc71ffff640 1 -- 172.21.15.77:0/3448525602 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc72c0109a0 con 0x7fc738102a20 2024-03-16T09:45:56.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.102+0000 7fc71ffff640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc7200a2800 0x7fc7200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.102+0000 7fc736ffd640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc7200a2800 0x7fc7200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.103+0000 7fc71ffff640 1 -- 172.21.15.77:0/3448525602 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fc72c0ce5f0 con 0x7fc738102a20 2024-03-16T09:45:56.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.103+0000 7fc736ffd640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc7200a2800 0x7fc7200a4cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc738102880 tx=0x7fc728009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:56.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.107+0000 7fc71ffff640 1 -- 172.21.15.77:0/3448525602 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc72c009de0 con 0x7fc738102a20 2024-03-16T09:45:56.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.322+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc73802d1b0 con 0x7fc738102a20 2024-03-16T09:45:56.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.322+0000 7fc71ffff640 1 -- 172.21.15.77:0/3448525602 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7fc72c09b050 con 0x7fc738102a20 2024-03-16T09:45:56.325 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:56.326 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1282436458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":1282436458},{"type":"v1","addr":"172.21.15.77:6835","nonce":1282436458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3058701505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3058701505},{"type":"v1","addr":"172.21.15.77:6837","nonce":3058701505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/1432550583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":1432550583},{"type":"v1","addr":"172.21.15.77:6839","nonce":1432550583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/3585324919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":3585324919},{"type":"v1","addr":"172.21.15.77:6841","nonce":3585324919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3738546284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3738546284},{"type":"v1","addr":"172.21.15.129:6833","nonce":3738546284}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3136672591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3136672591},{"type":"v1","addr":"172.21.15.119:6833","nonce":3136672591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T09:39:55.509822+0000","modified":"2024-03-16T09:41:49.295639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24503,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6835/2328435031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2328435031},{"type":"v1","addr":"172.21.15.129:6835","nonce":2328435031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.129:6837/814033341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":814033341},{"type":"v1","addr":"172.21.15.129:6837","nonce":814033341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/296042620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":296042620},{"type":"v1","addr":"172.21.15.119:6835","nonce":296042620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6839/1514269525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":1514269525},{"type":"v1","addr":"172.21.15.129:6839","nonce":1514269525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1948234265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1948234265},{"type":"v1","addr":"172.21.15.119:6837","nonce":1948234265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3029827381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3029827381},{"type":"v1","addr":"172.21.15.119:6839","nonce":3029827381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24481,24503,24545]},"id":1}]} 2024-03-16T09:45:56.326 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2024-03-16T09:45:56.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.324+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc7200a2800 msgr2=0x7fc7200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.324+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc7200a2800 0x7fc7200a4cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc738102880 tx=0x7fc728009500 comp rx=0 tx=0).stop 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc738102a20 msgr2=0x7fc738197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc738102a20 0x7fc738197f20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00d950 tx=0x7fc72c00de10 comp rx=0 tx=0).stop 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 shutdown_connections 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fc7200a2800 0x7fc7200a4cc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7381033e0 0x7fc73819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc738102a20 0x7fc738197f20 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 --2- 172.21.15.77:0/3448525602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc738101830 0x7fc7381979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 >> 172.21.15.77:0/3448525602 conn(0x7fc7380fcd80 msgr2=0x7fc738105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 shutdown_connections 2024-03-16T09:45:56.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.325+0000 7fc73d2ae640 1 -- 172.21.15.77:0/3448525602 wait complete. 2024-03-16T09:45:56.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T09:45:56.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 -- 172.21.15.77:0/1181455426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540ffe90 msgr2=0x7fac541002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 --2- 172.21.15.77:0/1181455426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540ffe90 0x7fac541002b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fac44009900 tx=0x7fac4402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:56.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 -- 172.21.15.77:0/1181455426 shutdown_connections 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 --2- 172.21.15.77:0/1181455426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac541019a0 0x7fac541082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 --2- 172.21.15.77:0/1181455426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac54100fe0 0x7fac54101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 --2- 172.21.15.77:0/1181455426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac540ffe90 0x7fac541002b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 -- 172.21.15.77:0/1181455426 >> 172.21.15.77:0/1181455426 conn(0x7fac540fb400 msgr2=0x7fac540fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 -- 172.21.15.77:0/1181455426 shutdown_connections 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.438+0000 7fac5a827640 1 -- 172.21.15.77:0/1181455426 wait complete. 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 Processor -- start 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 -- start start 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac540ffe90 0x7fac54197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac54100fe0 0x7fac54197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac541019a0 0x7fac5419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410b170 con 0x7fac541019a0 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410aff0 con 0x7fac54100fe0 2024-03-16T09:45:56.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac5a827640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410b2f0 con 0x7fac540ffe90 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac541019a0 0x7fac5419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac541019a0 0x7fac5419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56158/0 (socket says 172.21.15.77:56158) 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 -- 172.21.15.77:0/1047371606 learned_addr learned my addr 172.21.15.77:0/1047371606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac53fff640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac540ffe90 0x7fac54197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac537fe640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac54100fe0 0x7fac54197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 -- 172.21.15.77:0/1047371606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac540ffe90 msgr2=0x7fac54197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac540ffe90 0x7fac54197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 -- 172.21.15.77:0/1047371606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac54100fe0 msgr2=0x7fac54197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac54100fe0 0x7fac54197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.439+0000 7fac58dc4640 1 -- 172.21.15.77:0/1047371606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5419f6a0 con 0x7fac541019a0 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.440+0000 7fac58dc4640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac541019a0 0x7fac5419f030 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00eb10 tx=0x7fac4c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.440+0000 7fac517fa640 1 -- 172.21.15.77:0/1047371606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c004b20 con 0x7fac541019a0 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.440+0000 7fac517fa640 1 -- 172.21.15.77:0/1047371606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac4c004cc0 con 0x7fac541019a0 2024-03-16T09:45:56.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.440+0000 7fac517fa640 1 -- 172.21.15.77:0/1047371606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c010690 con 0x7fac541019a0 2024-03-16T09:45:56.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.440+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5419f990 con 0x7fac541019a0 2024-03-16T09:45:56.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.440+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5419fee0 con 0x7fac541019a0 2024-03-16T09:45:56.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.441+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac541a01d0 con 0x7fac541019a0 2024-03-16T09:45:56.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.442+0000 7fac517fa640 1 -- 172.21.15.77:0/1047371606 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fac4c005020 con 0x7fac541019a0 2024-03-16T09:45:56.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.442+0000 7fac517fa640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.443+0000 7fac53fff640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.443+0000 7fac517fa640 1 -- 172.21.15.77:0/1047371606 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fac4c0cf110 con 0x7fac541019a0 2024-03-16T09:45:56.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.443+0000 7fac53fff640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fac44009230 tx=0x7fac44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:56.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.447+0000 7fac517fa640 1 -- 172.21.15.77:0/1047371606 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac4c09c1c0 con 0x7fac541019a0 2024-03-16T09:45:56.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.624+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac541a03a0 con 0x7fac541019a0 2024-03-16T09:45:56.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.625+0000 7fac517fa640 1 -- 172.21.15.77:0/1047371606 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7fac4c09bac0 con 0x7fac541019a0 2024-03-16T09:45:56.629 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:45:56.629 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"e287e1d6-e377-11ee-95c9-87774f69a715","created":"2024-03-16T09:32:02.879385+0000","modified":"2024-03-16T09:40:58.857371+0000","last_up_change":"2024-03-16T09:39:06.601869+0000","last_in_change":"2024-03-16T09:38:45.339224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T09:35:24.036758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T09:39:54.014341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T09:39:54.733977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6e86c30-0964-4336-a711-17623a2896cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6803","nonce":3264923905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6805","nonce":3264923905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6809","nonce":3264923905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3264923905},{"type":"v1","addr":"172.21.15.77:6807","nonce":3264923905}]},"public_addr":"172.21.15.77:6803/3264923905","cluster_addr":"172.21.15.77:6805/3264923905","heartbeat_back_addr":"172.21.15.77:6809/3264923905","heartbeat_front_addr":"172.21.15.77:6807/3264923905","state":["exists","up"]},{"osd":1,"uuid":"03eeab1c-5a6a-4a70-a48d-df6a2d9183cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6801","nonce":2451540128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6803","nonce":2451540128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6807","nonce":2451540128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2451540128},{"type":"v1","addr":"172.21.15.119:6805","nonce":2451540128}]},"public_addr":"172.21.15.119:6801/2451540128","cluster_addr":"172.21.15.119:6803/2451540128","heartbeat_back_addr":"172.21.15.119:6807/2451540128","heartbeat_front_addr":"172.21.15.119:6805/2451540128","state":["exists","up"]},{"osd":2,"uuid":"b7e55cef-7750-4ba5-ab01-45d97769a4fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6801","nonce":2888466595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6803","nonce":2888466595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6807","nonce":2888466595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2888466595},{"type":"v1","addr":"172.21.15.129:6805","nonce":2888466595}]},"public_addr":"172.21.15.129:6801/2888466595","cluster_addr":"172.21.15.129:6803/2888466595","heartbeat_back_addr":"172.21.15.129:6807/2888466595","heartbeat_front_addr":"172.21.15.129:6805/2888466595","state":["exists","up"]},{"osd":3,"uuid":"d3207723-0545-48aa-8c7d-468dbc1b316a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6811","nonce":698693547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6813","nonce":698693547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6817","nonce":698693547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":698693547},{"type":"v1","addr":"172.21.15.77:6815","nonce":698693547}]},"public_addr":"172.21.15.77:6811/698693547","cluster_addr":"172.21.15.77:6813/698693547","heartbeat_back_addr":"172.21.15.77:6817/698693547","heartbeat_front_addr":"172.21.15.77:6815/698693547","state":["exists","up"]},{"osd":4,"uuid":"43d4adbe-baa5-4c4e-9a4a-d7edaf2011cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6809","nonce":3108480529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6811","nonce":3108480529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6815","nonce":3108480529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3108480529},{"type":"v1","addr":"172.21.15.119:6813","nonce":3108480529}]},"public_addr":"172.21.15.119:6809/3108480529","cluster_addr":"172.21.15.119:6811/3108480529","heartbeat_back_addr":"172.21.15.119:6815/3108480529","heartbeat_front_addr":"172.21.15.119:6813/3108480529","state":["exists","up"]},{"osd":5,"uuid":"f0413402-bee8-4802-8b4a-d3a0a48049a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6809","nonce":2429107441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6811","nonce":2429107441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6815","nonce":2429107441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2429107441},{"type":"v1","addr":"172.21.15.129:6813","nonce":2429107441}]},"public_addr":"172.21.15.129:6809/2429107441","cluster_addr":"172.21.15.129:6811/2429107441","heartbeat_back_addr":"172.21.15.129:6815/2429107441","heartbeat_front_addr":"172.21.15.129:6813/2429107441","state":["exists","up"]},{"osd":6,"uuid":"a6700df2-9147-4ff5-93f7-105320d2a596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6819","nonce":2647291293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6821","nonce":2647291293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6825","nonce":2647291293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2647291293},{"type":"v1","addr":"172.21.15.77:6823","nonce":2647291293}]},"public_addr":"172.21.15.77:6819/2647291293","cluster_addr":"172.21.15.77:6821/2647291293","heartbeat_back_addr":"172.21.15.77:6825/2647291293","heartbeat_front_addr":"172.21.15.77:6823/2647291293","state":["exists","up"]},{"osd":7,"uuid":"f717036c-9f74-49f7-9e6a-5a220593a044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6817","nonce":2297351860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6819","nonce":2297351860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6823","nonce":2297351860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2297351860},{"type":"v1","addr":"172.21.15.119:6821","nonce":2297351860}]},"public_addr":"172.21.15.119:6817/2297351860","cluster_addr":"172.21.15.119:6819/2297351860","heartbeat_back_addr":"172.21.15.119:6823/2297351860","heartbeat_front_addr":"172.21.15.119:6821/2297351860","state":["exists","up"]},{"osd":8,"uuid":"02a6c20f-3922-494c-a45a-84173f7c41c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6817","nonce":3565481849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6819","nonce":3565481849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6823","nonce":3565481849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3565481849},{"type":"v1","addr":"172.21.15.129:6821","nonce":3565481849}]},"public_addr":"172.21.15.129:6817/3565481849","cluster_addr":"172.21.15.129:6819/3565481849","heartbeat_back_addr":"172.21.15.129:6823/3565481849","heartbeat_front_addr":"172.21.15.129:6821/3565481849","state":["exists","up"]},{"osd":9,"uuid":"8e592022-204a-4db0-b470-63978907639a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6827","nonce":1881713362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6829","nonce":1881713362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6833","nonce":1881713362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":1881713362},{"type":"v1","addr":"172.21.15.77:6831","nonce":1881713362}]},"public_addr":"172.21.15.77:6827/1881713362","cluster_addr":"172.21.15.77:6829/1881713362","heartbeat_back_addr":"172.21.15.77:6833/1881713362","heartbeat_front_addr":"172.21.15.77:6831/1881713362","state":["exists","up"]},{"osd":10,"uuid":"68f623f5-73ff-46d2-8ff8-30767522cc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6825","nonce":1354177519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6827","nonce":1354177519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6831","nonce":1354177519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1354177519},{"type":"v1","addr":"172.21.15.119:6829","nonce":1354177519}]},"public_addr":"172.21.15.119:6825/1354177519","cluster_addr":"172.21.15.119:6827/1354177519","heartbeat_back_addr":"172.21.15.119:6831/1354177519","heartbeat_front_addr":"172.21.15.119:6829/1354177519","state":["exists","up"]},{"osd":11,"uuid":"ef748c28-d638-4d1d-a084-080ef7953c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6825","nonce":913848880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6827","nonce":913848880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6831","nonce":913848880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":913848880},{"type":"v1","addr":"172.21.15.129:6829","nonce":913848880}]},"public_addr":"172.21.15.129:6825/913848880","cluster_addr":"172.21.15.129:6827/913848880","heartbeat_back_addr":"172.21.15.129:6831/913848880","heartbeat_front_addr":"172.21.15.129:6829/913848880","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:32.229634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:34:56.841862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:21.375387+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:35:45.786961+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:10.487381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:34.305686+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:36:59.010895+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:23.363153+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:37:48.058581+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:13.306977+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:38:38.345491+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T09:39:03.685638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1569662437":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3524537702":"2024-03-17T09:32:57.693064+0000","172.21.15.77:6801/3741578273":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/2291840305":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2778231529":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6801/262452209":"2024-03-17T09:32:29.772291+0000","172.21.15.77:0/2049888475":"2024-03-17T09:32:57.693064+0000","172.21.15.77:0/3919406383":"2024-03-17T09:32:29.772291+0000","172.21.15.77:6800/3741578273":"2024-03-17T09:32:57.693064+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T09:45:56.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac3c0a2800 msgr2=0x7fac3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fac44009230 tx=0x7fac44005fd0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac541019a0 msgr2=0x7fac5419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac541019a0 0x7fac5419f030 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00eb10 tx=0x7fac4c00c2d0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 shutdown_connections 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac541019a0 0x7fac5419f030 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac54100fe0 0x7fac54197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 --2- 172.21.15.77:0/1047371606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac540ffe90 0x7fac54197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 >> 172.21.15.77:0/1047371606 conn(0x7fac540fb400 msgr2=0x7fac54106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 shutdown_connections 2024-03-16T09:45:56.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.627+0000 7fac5a827640 1 -- 172.21.15.77:0/1047371606 wait complete. 2024-03-16T09:45:56.642 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 -- 172.21.15.77:0/142524078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 msgr2=0x7f85c0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 --2- 172.21.15.77:0/142524078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0100b10 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009900 tx=0x7f85b402bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 -- 172.21.15.77:0/142524078 shutdown_connections 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 --2- 172.21.15.77:0/142524078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c01022a0 0x7f85c0108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 --2- 172.21.15.77:0/142524078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c01018e0 0x7f85c0101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 --2- 172.21.15.77:0/142524078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0100b10 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 -- 172.21.15.77:0/142524078 >> 172.21.15.77:0/142524078 conn(0x7f85c00fbc80 msgr2=0x7f85c00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 -- 172.21.15.77:0/142524078 shutdown_connections 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 -- 172.21.15.77:0/142524078 wait complete. 2024-03-16T09:45:56.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.741+0000 7f85c513b640 1 Processor -- start 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 -- start start 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c01018e0 0x7f85c0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c01022a0 0x7f85c019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85c010c1f0 con 0x7f85c01006f0 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f85c010c070 con 0x7f85c01022a0 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85c010c370 con 0x7f85c01018e0 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56172/0 (socket says 172.21.15.77:56172) 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 -- 172.21.15.77:0/2790151494 learned_addr learned my addr 172.21.15.77:0/2790151494 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bf59e640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c01022a0 0x7f85c019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85be59c640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c01018e0 0x7f85c0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 -- 172.21.15.77:0/2790151494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c01018e0 msgr2=0x7f85c0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c01018e0 0x7f85c0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 -- 172.21.15.77:0/2790151494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c01022a0 msgr2=0x7f85c019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c01022a0 0x7f85c019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 -- 172.21.15.77:0/2790151494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c019f6c0 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85bed9d640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0197aa0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f85b4000c00 tx=0x7f85b4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85a7fff640 1 -- 172.21.15.77:0/2790151494 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4043020 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c019f950 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85a7fff640 1 -- 172.21.15.77:0/2790151494 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b4004350 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c019fef0 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.742+0000 7f85a7fff640 1 -- 172.21.15.77:0/2790151494 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4003b20 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.743+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85c01a00c0 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.744+0000 7f85a7fff640 1 -- 172.21.15.77:0/2790151494 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f85b4046020 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.745+0000 7f85a7fff640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.745+0000 7f85a7fff640 1 -- 172.21.15.77:0/2790151494 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f85b40f0b60 con 0x7f85c01006f0 2024-03-16T09:45:56.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.745+0000 7f85be59c640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:56.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.745+0000 7f85be59c640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85a80a2800 0x7f85a80a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f85c0198fb0 tx=0x7f85b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:56.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.749+0000 7f85a7fff640 1 -- 172.21.15.77:0/2790151494 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f85b404d020 con 0x7f85c01006f0 2024-03-16T09:45:56.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.964+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f85c01a0290 con 0x7f85c01006f0 2024-03-16T09:45:56.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.966+0000 7f85a7fff640 1 -- 172.21.15.77:0/2790151494 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) ==== 161+0+0 (secure 0 0 0) 0x7f85b40bdc10 con 0x7f85c01006f0 2024-03-16T09:45:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-16T09:45:56.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85a80a2800 msgr2=0x7f85a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85a80a2800 0x7f85a80a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f85c0198fb0 tx=0x7f85b0005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:56.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 msgr2=0x7f85c0197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:56.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0197aa0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f85b4000c00 tx=0x7f85b4005dc0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 shutdown_connections 2024-03-16T09:45:56.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f85a80a2800 0x7f85a80a4cc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85c01022a0 0x7f85c019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85c01018e0 0x7f85c0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 --2- 172.21.15.77:0/2790151494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85c01006f0 0x7f85c0197aa0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:56.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.968+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 >> 172.21.15.77:0/2790151494 conn(0x7f85c00fbc80 msgr2=0x7f85c0069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:56.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.969+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 shutdown_connections 2024-03-16T09:45:56.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:56.969+0000 7f85c513b640 1 -- 172.21.15.77:0/2790151494 wait complete. 2024-03-16T09:45:56.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.085+0000 7f94d8937640 1 -- 172.21.15.77:0/203819055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 msgr2=0x7f94d4101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.085+0000 7f94d8937640 1 --2- 172.21.15.77:0/203819055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4101d50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009900 tx=0x7f94c802bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.085+0000 7f94d8937640 1 -- 172.21.15.77:0/203819055 shutdown_connections 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.085+0000 7f94d8937640 1 --2- 172.21.15.77:0/203819055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d4102290 0x7f94d4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.085+0000 7f94d8937640 1 --2- 172.21.15.77:0/203819055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4101d50 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.085+0000 7f94d8937640 1 --2- 172.21.15.77:0/203819055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94d41006e0 0x7f94d4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.085+0000 7f94d8937640 1 -- 172.21.15.77:0/203819055 >> 172.21.15.77:0/203819055 conn(0x7f94d40fbc70 msgr2=0x7f94d40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 -- 172.21.15.77:0/203819055 shutdown_connections 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 -- 172.21.15.77:0/203819055 wait complete. 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 Processor -- start 2024-03-16T09:45:57.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 -- start start 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d41006e0 0x7f94d4197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94d4102290 0x7f94d419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f94d410d330 con 0x7f94d41018d0 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f94d410d1b0 con 0x7f94d4102290 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d8937640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94d410d4b0 con 0x7f94d41006e0 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.086+0000 7f94d1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d41006e0 0x7f94d4197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56186/0 (socket says 172.21.15.77:56186) 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 -- 172.21.15.77:0/1980654888 learned_addr learned my addr 172.21.15.77:0/1980654888 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d2d9d640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94d4102290 0x7f94d419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 -- 172.21.15.77:0/1980654888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d41006e0 msgr2=0x7f94d4197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d41006e0 0x7f94d4197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 -- 172.21.15.77:0/1980654888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94d4102290 msgr2=0x7f94d419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94d4102290 0x7f94d419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 -- 172.21.15.77:0/1980654888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d419f630 con 0x7f94d41018d0 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d1d9b640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4197f50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f94c8006010 tx=0x7f94c8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94c37fe640 1 -- 172.21.15.77:0/1980654888 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8043020 con 0x7f94d41018d0 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d419f8c0 con 0x7f94d41018d0 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94c37fe640 1 -- 172.21.15.77:0/1980654888 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94c80043d0 con 0x7f94d41018d0 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94c37fe640 1 -- 172.21.15.77:0/1980654888 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8003b80 con 0x7f94d41018d0 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.087+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d419fe60 con 0x7f94d41018d0 2024-03-16T09:45:57.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.088+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d4107aa0 con 0x7f94d41018d0 2024-03-16T09:45:57.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.089+0000 7f94c37fe640 1 -- 172.21.15.77:0/1980654888 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f94c8046020 con 0x7f94d41018d0 2024-03-16T09:45:57.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.090+0000 7f94c37fe640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:57.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.090+0000 7f94d259c640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:57.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.090+0000 7f94c37fe640 1 -- 172.21.15.77:0/1980654888 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f94c80f1350 con 0x7f94d41018d0 2024-03-16T09:45:57.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.090+0000 7f94d259c640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f94d4101730 tx=0x7f94c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.093+0000 7f94c37fe640 1 -- 172.21.15.77:0/1980654888 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f94c804d020 con 0x7f94d41018d0 2024-03-16T09:45:57.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e[65415]: ignoring --setuser ceph since I am not root 2024-03-16T09:45:57.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e[65415]: ignoring --setgroup ceph since I am not root 2024-03-16T09:45:57.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mds[65419]: -- 172.21.15.119:0/1137166270 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c1e45d1a40 con 0x55c1e45a5000 2024-03-16T09:45:57.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e[65415]: starting mds.e at 2024-03-16T09:45:57.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:56 smithi119 ceph-mon[35632]: pgmap v467: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:56 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2060936674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:56 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3448525602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:56 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1047371606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: ignoring --setuser ceph since I am not root 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: ignoring --setgroup ceph since I am not root 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mds[66047]: -- 172.21.15.119:0/1643622357 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5613d7e0da40 con 0x5613d7de1000 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: starting mds.b at 2024-03-16T09:45:57.191 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h[65720]: ignoring --setuser ceph since I am not root 2024-03-16T09:45:57.192 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h[65720]: ignoring --setgroup ceph since I am not root 2024-03-16T09:45:57.192 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mds[65724]: -- 172.21.15.119:0/2325796159 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b08bd27a40 con 0x55b08bcfac00 2024-03-16T09:45:57.192 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h[65720]: starting mds.h at 2024-03-16T09:45:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:56 smithi077 ceph-mon[32171]: pgmap v467: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:45:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:56 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2060936674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:56 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3448525602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:56 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1047371606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a[32167]: 2024-03-16T09:45:56.964+0000 7f2a056ce700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:56 smithi129 ceph-mon[35841]: pgmap v467: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:56 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2060936674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:56 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3448525602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:56 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1047371606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c[66464]: ignoring --setuser ceph since I am not root 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c[66464]: ignoring --setgroup ceph since I am not root 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mds[66468]: -- 172.21.15.129:0/259698505 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d46d8dfa40 con 0x55d46d8b3000 2024-03-16T09:45:57.301 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c[66464]: starting mds.c at 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i[66771]: ignoring --setuser ceph since I am not root 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i[66771]: ignoring --setgroup ceph since I am not root 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mds[66775]: -- 172.21.15.129:0/4016138712 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5632c5fd7a40 con 0x5632c5faa800 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i[66771]: starting mds.i at 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f[66155]: ignoring --setuser ceph since I am not root 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f[66155]: ignoring --setgroup ceph since I am not root 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mds[66159]: -- 172.21.15.129:0/1154336961 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56172cd83a40 con 0x56172cd56c00 2024-03-16T09:45:57.302 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f[66155]: starting mds.f at 2024-03-16T09:45:57.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.308+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f94d402d1b0 con 0x7f94d41018d0 2024-03-16T09:45:57.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.972+0000 7f94c37fe640 1 -- 172.21.15.77:0/1980654888 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7f94c80be300 con 0x7f94d41018d0 2024-03-16T09:45:57.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f94bc0a2800 msgr2=0x7f94bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:57.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f94d4101730 tx=0x7f94c4005f90 comp rx=0 tx=0).stop 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 msgr2=0x7f94d4197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4197f50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f94c8006010 tx=0x7f94c8005e40 comp rx=0 tx=0).stop 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 shutdown_connections 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94d4102290 0x7f94d419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d41018d0 0x7f94d4197f50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 --2- 172.21.15.77:0/1980654888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94d41006e0 0x7f94d4197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 >> 172.21.15.77:0/1980654888 conn(0x7f94d40fbc70 msgr2=0x7f94d4104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 shutdown_connections 2024-03-16T09:45:57.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:57.975+0000 7f94d8937640 1 -- 172.21.15.77:0/1980654888 wait complete. 2024-03-16T09:45:57.993 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-16T09:45:58.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 -- 172.21.15.77:0/2800524377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0341017e0 msgr2=0x7fb034101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/2800524377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0341017e0 0x7fb034101c00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009ca0 tx=0x7fb02c02c0f0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 -- 172.21.15.77:0/2800524377 shutdown_connections 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/2800524377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb034103390 0x7fb034109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/2800524377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0341029d0 0x7fb034102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/2800524377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0341017e0 0x7fb034101c00 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 -- 172.21.15.77:0/2800524377 >> 172.21.15.77:0/2800524377 conn(0x7fb0340fcd50 msgr2=0x7fb0340ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 -- 172.21.15.77:0/2800524377 shutdown_connections 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 -- 172.21.15.77:0/2800524377 wait complete. 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.101+0000 7fb03bc6e640 1 Processor -- start 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 -- start start 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0341017e0 0x7fb034197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0341029d0 0x7fb034197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:58.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb034103390 0x7fb03419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb03410d360 con 0x7fb034103390 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb03410d1e0 con 0x7fb0341029d0 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb03410d4e0 con 0x7fb0341017e0 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb034103390 0x7fb03419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb034103390 0x7fb03419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56202/0 (socket says 172.21.15.77:56202) 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 -- 172.21.15.77:0/3622631095 learned_addr learned my addr 172.21.15.77:0/3622631095 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb039a0a640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0341017e0 0x7fb034197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb039209640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0341029d0 0x7fb034197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 -- 172.21.15.77:0/3622631095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0341017e0 msgr2=0x7fb034197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0341017e0 0x7fb034197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 -- 172.21.15.77:0/3622631095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0341029d0 msgr2=0x7fb034197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0341029d0 0x7fb034197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 -- 172.21.15.77:0/3622631095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03419f6b0 con 0x7fb034103390 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb039a0a640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0341017e0 0x7fb034197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb039209640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0341029d0 0x7fb034197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03a20b640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb034103390 0x7fb03419f040 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb030009fd0 tx=0x7fb03000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb02affd640 1 -- 172.21.15.77:0/3622631095 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030011830 con 0x7fb034103390 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb02affd640 1 -- 172.21.15.77:0/3622631095 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb030016070 con 0x7fb034103390 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb03419f9a0 con 0x7fb034103390 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.102+0000 7fb02affd640 1 -- 172.21.15.77:0/3622631095 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030010de0 con 0x7fb034103390 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.103+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb03419ff70 con 0x7fb034103390 2024-03-16T09:45:58.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.103+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb034108b70 con 0x7fb034103390 2024-03-16T09:45:58.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.104+0000 7fb02affd640 1 -- 172.21.15.77:0/3622631095 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb0300119d0 con 0x7fb034103390 2024-03-16T09:45:58.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.105+0000 7fb02affd640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0200a2800 0x7fb0200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:58.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.105+0000 7fb039a0a640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0200a2800 0x7fb0200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:58.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.105+0000 7fb02affd640 1 -- 172.21.15.77:0/3622631095 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb0300ce2e0 con 0x7fb034103390 2024-03-16T09:45:58.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.105+0000 7fb039a0a640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0200a2800 0x7fb0200a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb02c02c700 tx=0x7fb02c0324f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:58.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.110+0000 7fb02affd640 1 -- 172.21.15.77:0/3622631095 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb03009b290 con 0x7fb034103390 2024-03-16T09:45:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2790151494' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T09:45:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mon[35632]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T09:45:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mon[35632]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T09:45:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/2790151494' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T09:45:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mon[35632]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T09:45:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mon[35632]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T09:45:58.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:57 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1980654888' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:45:58.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2790151494' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T09:45:58.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:57 smithi077 ceph-mon[32171]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T09:45:58.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:57 smithi077 ceph-mon[32171]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T09:45:58.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/2790151494' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T09:45:58.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:57 smithi077 ceph-mon[32171]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T09:45:58.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:57 smithi077 ceph-mon[32171]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T09:45:58.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:57 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1980654888' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:45:58.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.285+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb03402d1b0 con 0x7fb034103390 2024-03-16T09:45:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2790151494' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T09:45:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mon[35841]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T09:45:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mon[35841]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T09:45:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/2790151494' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T09:45:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mon[35841]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T09:45:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mon[35841]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T09:45:58.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:57 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1980654888' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T09:45:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.975+0000 7fb02affd640 1 -- 172.21.15.77:0/3622631095 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) ==== 181+0+0 (secure 0 0 0) 0x7fb03009ab90 con 0x7fb034103390 2024-03-16T09:45:58.979 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-16T09:45:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0200a2800 msgr2=0x7fb0200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0200a2800 0x7fb0200a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb02c02c700 tx=0x7fb02c0324f0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb034103390 msgr2=0x7fb03419f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb034103390 0x7fb03419f040 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb030009fd0 tx=0x7fb03000b8e0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 shutdown_connections 2024-03-16T09:45:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7fb0200a2800 0x7fb0200a4cc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb034103390 0x7fb03419f040 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0341029d0 0x7fb034197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 --2- 172.21.15.77:0/3622631095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0341017e0 0x7fb034197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 >> 172.21.15.77:0/3622631095 conn(0x7fb0340fcd50 msgr2=0x7fb034105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 shutdown_connections 2024-03-16T09:45:58.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:58.979+0000 7fb03bc6e640 1 -- 172.21.15.77:0/3622631095 wait complete. 2024-03-16T09:45:58.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 -- 172.21.15.77:0/3140991803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801006f0 msgr2=0x7f0180100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 --2- 172.21.15.77:0/3140991803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801006f0 0x7f0180100b10 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0170009900 tx=0x7f017002bc30 comp rx=0 tx=0).stop 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 -- 172.21.15.77:0/3140991803 shutdown_connections 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 --2- 172.21.15.77:0/3140991803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01801022a0 0x7f0180108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 --2- 172.21.15.77:0/3140991803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01801018e0 0x7f0180101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 --2- 172.21.15.77:0/3140991803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801006f0 0x7f0180100b10 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 -- 172.21.15.77:0/3140991803 >> 172.21.15.77:0/3140991803 conn(0x7f01800fbc80 msgr2=0x7f01800fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 -- 172.21.15.77:0/3140991803 shutdown_connections 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 -- 172.21.15.77:0/3140991803 wait complete. 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 Processor -- start 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 -- start start 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01801006f0 0x7f0180197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01801018e0 0x7f0180197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.099+0000 7f0185909640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801022a0 0x7f018019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0185909640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f018010d360 con 0x7f01801022a0 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0185909640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f018010d1e0 con 0x7f01801018e0 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0185909640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f018010d4e0 con 0x7f01801006f0 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801022a0 0x7f018019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801022a0 0x7f018019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56212/0 (socket says 172.21.15.77:56212) 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 -- 172.21.15.77:0/1587632140 learned_addr learned my addr 172.21.15.77:0/1587632140 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017effd640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01801006f0 0x7f0180197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:59.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017e7fc640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01801018e0 0x7f0180197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 -- 172.21.15.77:0/1587632140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01801006f0 msgr2=0x7f0180197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01801006f0 0x7f0180197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 -- 172.21.15.77:0/1587632140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01801018e0 msgr2=0x7f0180197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01801018e0 0x7f0180197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 -- 172.21.15.77:0/1587632140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018019f630 con 0x7f01801022a0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017e7fc640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01801018e0 0x7f0180197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f017f7fe640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801022a0 0x7f018019efc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f017800cce0 tx=0x7f017800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0184907640 1 -- 172.21.15.77:0/1587632140 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017800f450 con 0x7f01801022a0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0184907640 1 -- 172.21.15.77:0/1587632140 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f017800faa0 con 0x7f01801022a0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018019f920 con 0x7f01801022a0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0184907640 1 -- 172.21.15.77:0/1587632140 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178018b70 con 0x7f01801022a0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.100+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018019fe20 con 0x7f01801022a0 2024-03-16T09:45:59.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.101+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0180107a90 con 0x7f01801022a0 2024-03-16T09:45:59.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.102+0000 7f0184907640 1 -- 172.21.15.77:0/1587632140 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0178018d10 con 0x7f01801022a0 2024-03-16T09:45:59.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.103+0000 7f0184907640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f01680a2800 0x7f01680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T09:45:59.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.103+0000 7f017effd640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f01680a2800 0x7f01680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T09:45:59.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.103+0000 7f0184907640 1 -- 172.21.15.77:0/1587632140 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 7944+0+0 (secure 0 0 0) 0x7f01780ced20 con 0x7f01801022a0 2024-03-16T09:45:59.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.103+0000 7f017effd640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f01680a2800 0x7f01680a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f017002c240 tx=0x7f017002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T09:45:59.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.107+0000 7f0184907640 1 -- 172.21.15.77:0/1587632140 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0178014030 con 0x7f01801022a0 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: pgmap v469: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: Cluster is now healthy 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1980654888' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6837/2467560112,v1:172.21.15.129:6839/2467560112] up:boot 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6834/970998038,v1:172.21.15.129:6835/970998038] up:boot 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6836/3799987745,v1:172.21.15.119:6839/3799987745] up:boot 2024-03-16T09:45:59.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.129:6836/3923087257,v1:172.21.15.129:6838/3923087257] up:boot 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6835/1825185695,v1:172.21.15.119:6838/1825185695] up:boot 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: mds.? [v2:172.21.15.119:6834/594038916,v1:172.21.15.119:6837/594038916] up:boot 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: fsmap 12 up:standby 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T09:45:59.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:58 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3622631095' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T09:45:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: pgmap v469: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-16T09:45:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T09:45:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T09:45:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: Cluster is now healthy 2024-03-16T09:45:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1980654888' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T09:45:59.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6837/2467560112,v1:172.21.15.129:6839/2467560112] up:boot 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6834/970998038,v1:172.21.15.129:6835/970998038] up:boot 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6836/3799987745,v1:172.21.15.119:6839/3799987745] up:boot 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.129:6836/3923087257,v1:172.21.15.129:6838/3923087257] up:boot 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6835/1825185695,v1:172.21.15.119:6838/1825185695] up:boot 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: mds.? [v2:172.21.15.119:6834/594038916,v1:172.21.15.119:6837/594038916] up:boot 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: fsmap 12 up:standby 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T09:45:59.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:58 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3622631095' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T09:45:59.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.284+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f018002d1b0 con 0x7f01801022a0 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: pgmap v469: 97 pgs: 97 active+clean; 979 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: Cluster is now healthy 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1980654888' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6837/2467560112,v1:172.21.15.129:6839/2467560112] up:boot 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6834/970998038,v1:172.21.15.129:6835/970998038] up:boot 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6836/3799987745,v1:172.21.15.119:6839/3799987745] up:boot 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.129:6836/3923087257,v1:172.21.15.129:6838/3923087257] up:boot 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6835/1825185695,v1:172.21.15.119:6838/1825185695] up:boot 2024-03-16T09:45:59.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: mds.? [v2:172.21.15.119:6834/594038916,v1:172.21.15.119:6837/594038916] up:boot 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: fsmap 12 up:standby 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='mgr.14152 172.21.15.77:0/1379663073' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T09:45:59.302 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:58 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3622631095' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T09:45:59.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.981+0000 7f0184907640 1 -- 172.21.15.77:0/1587632140 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) ==== 169+0+0 (secure 0 0 0) 0x7f017809bf30 con 0x7f01801022a0 2024-03-16T09:45:59.985 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f01680a2800 msgr2=0x7f01680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f01680a2800 0x7f01680a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f017002c240 tx=0x7f017002c7e0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801022a0 msgr2=0x7f018019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801022a0 0x7f018019efc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f017800cce0 tx=0x7f017800a620 comp rx=0 tx=0).stop 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 shutdown_connections 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:6800/3114858521,v1:172.21.15.77:6801/3114858521] conn(0x7f01680a2800 0x7f01680a4cc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01801022a0 0x7f018019efc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01801018e0 0x7f0180197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 --2- 172.21.15.77:0/1587632140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01801006f0 0x7f0180197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T09:45:59.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 >> 172.21.15.77:0/1587632140 conn(0x7f01800fbc80 msgr2=0x7f0180104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T09:45:59.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 shutdown_connections 2024-03-16T09:45:59.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T09:45:59.984+0000 7f0185909640 1 -- 172.21.15.77:0/1587632140 wait complete. 2024-03-16T09:46:00.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:46:00.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:46:00.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:46:00.101 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-16T09:46:00.101 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T09:46:00.118 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T09:46:00.133 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T09:46:00.159 INFO:tasks.cephadm:Stopping all daemons... 2024-03-16T09:46:00.160 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-16T09:46:00.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a 2024-03-16T09:46:00.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:59 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/3622631095' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T09:46:00.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:59 smithi077 ceph-mon[32171]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T09:46:00.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:59 smithi077 ceph-mon[32171]: from='client.? 172.21.15.77:0/1587632140' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T09:46:00.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:45:59 smithi077 ceph-mon[32171]: pgmap v471: 65 pgs: 65 active+clean; 577 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:46:00.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:46:00 smithi077 systemd[1]: Stopping Ceph mon.a for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:59 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/3622631095' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T09:46:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:59 smithi129 ceph-mon[35841]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T09:46:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:59 smithi129 ceph-mon[35841]: from='client.? 172.21.15.77:0/1587632140' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T09:46:00.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:45:59 smithi129 ceph-mon[35841]: pgmap v471: 65 pgs: 65 active+clean; 577 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:46:00.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:59 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/3622631095' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T09:46:00.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:59 smithi119 ceph-mon[35632]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T09:46:00.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:59 smithi119 ceph-mon[35632]: from='client.? 172.21.15.77:0/1587632140' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T09:46:00.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:45:59 smithi119 ceph-mon[35632]: pgmap v471: 65 pgs: 65 active+clean; 577 KiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T09:46:00.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:46:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a[32167]: 2024-03-16T09:46:00.477+0000 7f2a0aed9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:00.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:46:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a[32167]: 2024-03-16T09:46:00.477+0000 7f2a0aed9700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-16T09:46:00.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:46:00 smithi077 podman[91534]: 2024-03-16 09:46:00.621284063 +0000 UTC m=+0.250862998 container died 80a2c43e44ea6f70a7a1b9f0990c41d0171778fa970bbbc1f6d3fd94ee4a5946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:46:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:46:01 smithi077 podman[91534]: 2024-03-16 09:46:01.103914083 +0000 UTC m=+0.733493018 container cleanup 80a2c43e44ea6f70a7a1b9f0990c41d0171778fa970bbbc1f6d3fd94ee4a5946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T09:46:01.259 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 09:46:01 smithi077 bash[91534]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-a 2024-03-16T09:46:01.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.a.service' 2024-03-16T09:46:01.576 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:01.576 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-16T09:46:01.576 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-16T09:46:01.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.b 2024-03-16T09:46:01.887 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:46:01 smithi119 systemd[1]: Stopping Ceph mon.b for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:46:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-b[35628]: 2024-03-16T09:46:01.884+0000 7f30129a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:46:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-b[35628]: 2024-03-16T09:46:01.884+0000 7f30129a9700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-16T09:46:02.190 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:46:01 smithi119 podman[69059]: 2024-03-16 09:46:01.966376538 +0000 UTC m=+0.191129263 container died 84d0a0b3dd423879946f1a0fba0ed0ffaa397281a529e82e34b8efe3f15ff706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-16T09:46:02.513 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:46:02 smithi119 podman[69059]: 2024-03-16 09:46:02.276238233 +0000 UTC m=+0.500990934 container cleanup 84d0a0b3dd423879946f1a0fba0ed0ffaa397281a529e82e34b8efe3f15ff706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-16T09:46:02.513 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 16 09:46:02 smithi119 bash[69059]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-b 2024-03-16T09:46:02.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.b.service' 2024-03-16T09:46:02.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:02.704 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-16T09:46:02.704 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-16T09:46:02.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.c 2024-03-16T09:46:03.005 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:46:02 smithi129 systemd[1]: Stopping Ceph mon.c for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:03.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:46:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c[35837]: 2024-03-16T09:46:03.002+0000 7f7d88c35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:03.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:46:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c[35837]: 2024-03-16T09:46:03.002+0000 7f7d88c35700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-16T09:46:03.301 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:46:03 smithi129 podman[69869]: 2024-03-16 09:46:03.166650642 +0000 UTC m=+0.272895940 container died a8d3da876c9cdca8c983f6eca9bbd97001dc004242f7761336b8ce035e26c9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:46:03.800 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:46:03 smithi129 podman[69869]: 2024-03-16 09:46:03.537072087 +0000 UTC m=+0.643317385 container cleanup a8d3da876c9cdca8c983f6eca9bbd97001dc004242f7761336b8ce035e26c9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T09:46:03.801 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 16 09:46:03 smithi129 bash[69869]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mon-c 2024-03-16T09:46:03.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mon.c.service' 2024-03-16T09:46:04.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:04.015 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-16T09:46:04.015 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-16T09:46:04.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x 2024-03-16T09:46:04.311 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:04 smithi077 systemd[1]: Stopping Ceph mgr.x for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:04.570 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:04 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:46:04.308+0000 7fe393fe9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:04.570 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:04 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x[32403]: 2024-03-16T09:46:04.308+0000 7fe393fe9700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-16T09:46:04.570 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:04 smithi077 podman[91656]: 2024-03-16 09:46:04.339877551 +0000 UTC m=+0.139979009 container died 4a5fa3c0118f7363784dc713cf1320011d7f2276301a744de6ba4757875e1ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-16T09:46:04.904 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:04 smithi077 podman[91656]: 2024-03-16 09:46:04.676082358 +0000 UTC m=+0.476183820 container cleanup 4a5fa3c0118f7363784dc713cf1320011d7f2276301a744de6ba4757875e1ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T09:46:04.904 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:04 smithi077 bash[91656]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x 2024-03-16T09:46:05.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-16T09:46:05.170 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:04 smithi077 podman[91670]: 2024-03-16 09:46:04.902450379 +0000 UTC m=+0.561007323 container remove 4a5fa3c0118f7363784dc713cf1320011d7f2276301a744de6ba4757875e1ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-16T09:46:05.170 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:05 smithi077 systemd[1]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x.service: Deactivated successfully. 2024-03-16T09:46:05.170 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:05 smithi077 systemd[1]: Stopped Ceph mgr.x for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:46:05.170 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 09:46:05 smithi077 systemd[1]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.x.service: Consumed 1min 993ms CPU time. 2024-03-16T09:46:05.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:05.184 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-16T09:46:05.184 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-16T09:46:05.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.y 2024-03-16T09:46:05.520 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:46:05 smithi119 systemd[1]: Stopping Ceph mgr.y for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:05.834 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:46:05 smithi119 podman[69180]: 2024-03-16 09:46:05.518246915 +0000 UTC m=+0.141594822 container died 60250e9ab4be2ecb8028971e7777c713dcaed4d9ecb3a02b37cf86b9c466bb61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-16T09:46:06.184 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:46:05 smithi119 podman[69180]: 2024-03-16 09:46:05.91469318 +0000 UTC m=+0.538041128 container cleanup 60250e9ab4be2ecb8028971e7777c713dcaed4d9ecb3a02b37cf86b9c466bb61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-16T09:46:06.184 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 16 09:46:05 smithi119 bash[69180]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-y 2024-03-16T09:46:06.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-16T09:46:06.357 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:06.357 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-16T09:46:06.357 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-16T09:46:06.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.z 2024-03-16T09:46:06.689 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:46:06 smithi129 systemd[1]: Stopping Ceph mgr.z for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:06.994 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:46:06 smithi129 podman[69993]: 2024-03-16 09:46:06.687245681 +0000 UTC m=+0.139655784 container died bc3aef4b02e461e840f80d4911b15b0560f759c08df6f8a722f663ffe9df0631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T09:46:07.301 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:46:07 smithi129 podman[69993]: 2024-03-16 09:46:07.091708048 +0000 UTC m=+0.544118151 container cleanup bc3aef4b02e461e840f80d4911b15b0560f759c08df6f8a722f663ffe9df0631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-16T09:46:07.301 INFO:journalctl@ceph.mgr.z.smithi129.stdout:Mar 16 09:46:07 smithi129 bash[69993]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mgr-z 2024-03-16T09:46:07.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-16T09:46:07.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:07.497 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-16T09:46:07.497 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-16T09:46:07.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.0 2024-03-16T09:46:07.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:07 smithi077 systemd[1]: Stopping Ceph osd.0 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:08.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0[47393]: 2024-03-16T09:46:07.796+0000 7f4566f09700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:08.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0[47393]: 2024-03-16T09:46:07.796+0000 7f4566f09700 -1 osd.0 91 *** Got signal Terminated *** 2024-03-16T09:46:08.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0[47393]: 2024-03-16T09:46:07.796+0000 7f4566f09700 -1 osd.0 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:46:13.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:13 smithi077 podman[91776]: 2024-03-16 09:46:13.116030244 +0000 UTC m=+5.439109692 container died e8d8bbc9ab7741060c1bd172968586bcbbfe178ac3da15c0a9eda07ac4521f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-16T09:46:13.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:13 smithi077 podman[91776]: 2024-03-16 09:46:13.581142699 +0000 UTC m=+5.904222186 container cleanup e8d8bbc9ab7741060c1bd172968586bcbbfe178ac3da15c0a9eda07ac4521f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T09:46:13.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:13 smithi077 bash[91776]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0 2024-03-16T09:46:14.194 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:13 smithi077 podman[91795]: 2024-03-16 09:46:13.85787427 +0000 UTC m=+0.740530912 container remove e8d8bbc9ab7741060c1bd172968586bcbbfe178ac3da15c0a9eda07ac4521f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-16T09:46:14.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:14 smithi077 podman[91884]: 2024-03-16 09:46:14.191950816 +0000 UTC m=+0.112780387 container create 4380934f59147a456068967e2df71e002eee97f5066a65eebac46a4a0c337754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-16T09:46:14.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:14 smithi077 podman[91884]: 2024-03-16 09:46:14.096912832 +0000 UTC m=+0.017742416 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:46:14.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:14 smithi077 podman[91884]: 2024-03-16 09:46:14.427488126 +0000 UTC m=+0.348317708 container init 4380934f59147a456068967e2df71e002eee97f5066a65eebac46a4a0c337754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-16T09:46:14.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:14 smithi077 podman[91884]: 2024-03-16 09:46:14.430652528 +0000 UTC m=+0.351482101 container start 4380934f59147a456068967e2df71e002eee97f5066a65eebac46a4a0c337754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-16T09:46:14.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:14 smithi077 podman[91884]: 2024-03-16 09:46:14.502536617 +0000 UTC m=+0.423366203 container attach 4380934f59147a456068967e2df71e002eee97f5066a65eebac46a4a0c337754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T09:46:14.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 09:46:14 smithi077 podman[91884]: 2024-03-16 09:46:14.723587459 +0000 UTC m=+0.644417022 container died 4380934f59147a456068967e2df71e002eee97f5066a65eebac46a4a0c337754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T09:46:15.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.0.service' 2024-03-16T09:46:15.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:15.377 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-16T09:46:15.377 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-16T09:46:15.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.1 2024-03-16T09:46:15.679 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:15 smithi119 systemd[1]: Stopping Ceph osd.1 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:15.940 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1[40195]: 2024-03-16T09:46:15.676+0000 7f7379469700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:15.940 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1[40195]: 2024-03-16T09:46:15.676+0000 7f7379469700 -1 osd.1 91 *** Got signal Terminated *** 2024-03-16T09:46:15.940 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1[40195]: 2024-03-16T09:46:15.676+0000 7f7379469700 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:46:21.413 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:21 smithi119 podman[69307]: 2024-03-16 09:46:21.115694346 +0000 UTC m=+5.547033988 container died 3bd8f03faeaa05d99b5da254fe451857e7c8b7147399581d0e65f6a6cbc6f2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-16T09:46:21.690 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:21 smithi119 podman[69307]: 2024-03-16 09:46:21.493579384 +0000 UTC m=+5.924919031 container cleanup 3bd8f03faeaa05d99b5da254fe451857e7c8b7147399581d0e65f6a6cbc6f2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-16T09:46:21.690 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:21 smithi119 bash[69307]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1 2024-03-16T09:46:22.190 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:21 smithi119 podman[69337]: 2024-03-16 09:46:21.937006098 +0000 UTC m=+0.813354576 container remove 3bd8f03faeaa05d99b5da254fe451857e7c8b7147399581d0e65f6a6cbc6f2fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:46:22.517 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:22 smithi119 podman[69432]: 2024-03-16 09:46:22.159648596 +0000 UTC m=+0.017367996 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:46:22.517 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:22 smithi119 podman[69432]: 2024-03-16 09:46:22.287887069 +0000 UTC m=+0.145606459 container create de3cce3c4e2b25cd3259532b177b56a69c22fe8a640bb9f9d75c89078a0b5046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-16T09:46:22.807 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:22 smithi119 podman[69432]: 2024-03-16 09:46:22.515200819 +0000 UTC m=+0.372920209 container init de3cce3c4e2b25cd3259532b177b56a69c22fe8a640bb9f9d75c89078a0b5046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T09:46:22.807 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:22 smithi119 podman[69432]: 2024-03-16 09:46:22.518512401 +0000 UTC m=+0.376231792 container start de3cce3c4e2b25cd3259532b177b56a69c22fe8a640bb9f9d75c89078a0b5046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T09:46:22.807 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:22 smithi119 podman[69432]: 2024-03-16 09:46:22.598755627 +0000 UTC m=+0.456475016 container attach de3cce3c4e2b25cd3259532b177b56a69c22fe8a640bb9f9d75c89078a0b5046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T09:46:23.136 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:22 smithi119 podman[69432]: 2024-03-16 09:46:22.805458164 +0000 UTC m=+0.663177569 container died de3cce3c4e2b25cd3259532b177b56a69c22fe8a640bb9f9d75c89078a0b5046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T09:46:23.365 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.1.service' 2024-03-16T09:46:23.392 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:23 smithi119 podman[69458]: 2024-03-16 09:46:23.335908494 +0000 UTC m=+0.524929130 container remove de3cce3c4e2b25cd3259532b177b56a69c22fe8a640bb9f9d75c89078a0b5046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:46:23.393 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:23 smithi119 systemd[1]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-16T09:46:23.393 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:23 smithi119 systemd[1]: Stopped Ceph osd.1 for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:46:23.393 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 16 09:46:23 smithi119 systemd[1]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.1.service: Consumed 23.444s CPU time. 2024-03-16T09:46:23.408 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:23.408 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-16T09:46:23.408 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-16T09:46:23.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.2 2024-03-16T09:46:23.715 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:23 smithi129 systemd[1]: Stopping Ceph osd.2 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:24.051 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2[40999]: 2024-03-16T09:46:23.712+0000 7f45fecac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:24.051 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2[40999]: 2024-03-16T09:46:23.712+0000 7f45fecac700 -1 osd.2 91 *** Got signal Terminated *** 2024-03-16T09:46:24.051 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2[40999]: 2024-03-16T09:46:23.712+0000 7f45fecac700 -1 osd.2 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:46:29.482 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:29 smithi129 podman[70132]: 2024-03-16 09:46:29.165251574 +0000 UTC m=+5.566090020 container died 8ffed0eb16cefe101777746bc24a306ce886eb66b2f8137aae48f88e6220c7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T09:46:29.798 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:29 smithi129 podman[70132]: 2024-03-16 09:46:29.562372976 +0000 UTC m=+5.963211431 container cleanup 8ffed0eb16cefe101777746bc24a306ce886eb66b2f8137aae48f88e6220c7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T09:46:29.799 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:29 smithi129 bash[70132]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2 2024-03-16T09:46:30.050 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:29 smithi129 podman[70149]: 2024-03-16 09:46:29.796844549 +0000 UTC m=+0.625084800 container remove 8ffed0eb16cefe101777746bc24a306ce886eb66b2f8137aae48f88e6220c7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T09:46:30.427 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:30 smithi129 podman[70240]: 2024-03-16 09:46:30.031085355 +0000 UTC m=+0.018253976 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:46:30.427 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:30 smithi129 podman[70240]: 2024-03-16 09:46:30.156074093 +0000 UTC m=+0.143242751 container create b2e9cbda3b666efcfe85ec9862f69ded9042ab1fb03c134323dcee735fb328af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:46:30.427 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:30 smithi129 podman[70240]: 2024-03-16 09:46:30.425649288 +0000 UTC m=+0.412817910 container init b2e9cbda3b666efcfe85ec9862f69ded9042ab1fb03c134323dcee735fb328af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-16T09:46:30.721 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:30 smithi129 podman[70240]: 2024-03-16 09:46:30.429230581 +0000 UTC m=+0.416399196 container start b2e9cbda3b666efcfe85ec9862f69ded9042ab1fb03c134323dcee735fb328af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T09:46:30.721 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:30 smithi129 podman[70240]: 2024-03-16 09:46:30.542396506 +0000 UTC m=+0.529565138 container attach b2e9cbda3b666efcfe85ec9862f69ded9042ab1fb03c134323dcee735fb328af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:46:31.014 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 16 09:46:30 smithi129 podman[70240]: 2024-03-16 09:46:30.719087299 +0000 UTC m=+0.706255913 container died b2e9cbda3b666efcfe85ec9862f69ded9042ab1fb03c134323dcee735fb328af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T09:46:31.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.2.service' 2024-03-16T09:46:31.352 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:31.352 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-16T09:46:31.352 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-16T09:46:31.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.3 2024-03-16T09:46:31.653 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:31 smithi077 systemd[1]: Stopping Ceph osd.3 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:31.915 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:31 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:46:31.650+0000 7f31a9b1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:31.915 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:31 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:46:31.650+0000 7f31a9b1a700 -1 osd.3 91 *** Got signal Terminated *** 2024-03-16T09:46:31.915 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:31 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:46:31.650+0000 7f31a9b1a700 -1 osd.3 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:46:34.009 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:33 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:46:33.689+0000 7f31a610e700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:07.516374+0000 front 2024-03-16T09:46:07.516316+0000 (oldest deadline 2024-03-16T09:46:33.416443+0000) 2024-03-16T09:46:34.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:34.183+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:34.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:34.110+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:34.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:34.156+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:35.009 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:34 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:46:34.646+0000 7f31a610e700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:07.516374+0000 front 2024-03-16T09:46:07.516316+0000 (oldest deadline 2024-03-16T09:46:33.416443+0000) 2024-03-16T09:46:35.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:34 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:34.585+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:35.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:35.034+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:35.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:35.202+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:35.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:35.162+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:35.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:35.155+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:36.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:35.577+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:36.009 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:35 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3[54210]: 2024-03-16T09:46:35.661+0000 7f31a610e700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:07.516374+0000 front 2024-03-16T09:46:07.516316+0000 (oldest deadline 2024-03-16T09:46:33.416443+0000) 2024-03-16T09:46:36.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:36 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:36.006+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:36.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:36.227+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:36.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:36.203+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:36.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:36.144+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:37.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:36 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:36.623+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:37.446 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:37 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:37.048+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:37.446 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:37 smithi077 podman[91993]: 2024-03-16 09:46:37.157908186 +0000 UTC m=+5.615899673 container died 77ac324b72e34aa6796465b26ce5133f8b8d3b87dc1166adf4c8e9911151c313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T09:46:37.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:37 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:37.269+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:37.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:37 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:37.232+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:37.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:37 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:37.123+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:37.759 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:37 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:37.582+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:37.759 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:37 smithi077 podman[91993]: 2024-03-16 09:46:37.618688829 +0000 UTC m=+6.076680383 container cleanup 77ac324b72e34aa6796465b26ce5133f8b8d3b87dc1166adf4c8e9911151c313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:46:37.759 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:37 smithi077 bash[91993]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3 2024-03-16T09:46:38.234 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:37 smithi077 podman[92020]: 2024-03-16 09:46:37.898933179 +0000 UTC m=+0.735822116 container remove 77ac324b72e34aa6796465b26ce5133f8b8d3b87dc1166adf4c8e9911151c313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T09:46:38.234 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:38 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:38.044+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:38.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:38 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:38.071+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:38.509 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:38 smithi077 podman[92113]: 2024-03-16 09:46:38.135405184 +0000 UTC m=+0.017452993 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:46:38.509 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:38 smithi077 podman[92113]: 2024-03-16 09:46:38.23746737 +0000 UTC m=+0.119515203 container create 3c1fb51888d171d1522e84a69cc86037cc6c7eda17ac52dae8931cc1788d31d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-16T09:46:38.509 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:38 smithi077 podman[92113]: 2024-03-16 09:46:38.473241137 +0000 UTC m=+0.355288927 container init 3c1fb51888d171d1522e84a69cc86037cc6c7eda17ac52dae8931cc1788d31d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-16T09:46:38.509 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:38 smithi077 podman[92113]: 2024-03-16 09:46:38.476678068 +0000 UTC m=+0.358725862 container start 3c1fb51888d171d1522e84a69cc86037cc6c7eda17ac52dae8931cc1788d31d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T09:46:38.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:38.235+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:38.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:38.196+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:38.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:38.154+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:38.773 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:38 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:38.569+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:38.773 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:38 smithi077 podman[92113]: 2024-03-16 09:46:38.565084027 +0000 UTC m=+0.447131821 container attach 3c1fb51888d171d1522e84a69cc86037cc6c7eda17ac52dae8931cc1788d31d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T09:46:39.039 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 09:46:38 smithi077 podman[92113]: 2024-03-16 09:46:38.771232224 +0000 UTC m=+0.653280021 container died 3c1fb51888d171d1522e84a69cc86037cc6c7eda17ac52dae8931cc1788d31d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:46:39.362 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:39 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:39.035+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:39.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.3.service' 2024-03-16T09:46:39.419 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:39.419 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-16T09:46:39.420 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-16T09:46:39.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.4 2024-03-16T09:46:39.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:39 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:39.074+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:39.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:39 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:39.233+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:39.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:39 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:39.224+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:39.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:39 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:39.146+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:39.725 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:39 smithi119 systemd[1]: Stopping Ceph osd.4 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:39.759 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:39 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:39.596+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:40.102 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:39 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4[46852]: 2024-03-16T09:46:39.720+0000 7f08a8d82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:40.102 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:39 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4[46852]: 2024-03-16T09:46:39.720+0000 7f08a8d82700 -1 osd.4 91 *** Got signal Terminated *** 2024-03-16T09:46:40.102 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:39 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4[46852]: 2024-03-16T09:46:39.720+0000 7f08a8d82700 -1 osd.4 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:46:40.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:40 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:40.100+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:40.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:40 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:40.025+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:40.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:40 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:40.222+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:40.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:40 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:40.239+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:40.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:40 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:40.185+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:41.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:40 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:40.615+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:41.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:41 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:41.116+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:41.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:41 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:41.031+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:41.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:41 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:41.235+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:41.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:41 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:41.228+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:41.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:41 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:41.226+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:42.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:41 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:41.617+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:42.259 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:42 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:42.195+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:42.352 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:42 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:42.061+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:42.352 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:42 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:42.061+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:42.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:42 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:42.140+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:42.550 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:42 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:42.257+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:42.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:42 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:42.261+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:42.759 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:42 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:42.582+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:42.759 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:42 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:42.582+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:43.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:43 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:43.021+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:43.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:43 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:43.021+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:43.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:43 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:43.129+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:43.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:43 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:43.179+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:43.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:43 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:43.179+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:43.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:43 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:43.284+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:43.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:43 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:43.284+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:43.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:43 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:43.260+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:43.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:43 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:43.260+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:44.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:43 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:43.623+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:44.010 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:43 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:43.623+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:44.143 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:43 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:43.797+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:44.419 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:44 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:44.140+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:44.419 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:44 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:44.140+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:44.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:44 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:44.051+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:44.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:44 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:44.051+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:44.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:44 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:44.188+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:44.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:44 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:44.188+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:44.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:44 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:44.249+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:44.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:44 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:44.249+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:44.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:44 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:44.220+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:44.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:44 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:44.220+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:45.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:44 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:44.587+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:45.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:44 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:44.587+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:45.119 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:44 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:44.836+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:45.440 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:45 smithi119 podman[69539]: 2024-03-16 09:46:45.11789189 +0000 UTC m=+5.509174018 container died 5f1ef8507d74cd722fb4b04edc36da0794a1a53f3194816c58c0787f06e6c636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-16T09:46:45.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:45 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:45.144+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:45.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:45 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:45.144+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:45.452 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:45 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:45.283+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:45.452 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:45 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:45.283+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:45.452 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:45 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:45.205+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:45.452 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:45 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:45.205+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:45.453 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:45 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:45.145+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:45.453 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:45 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:45.145+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:45.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:45.026+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:45.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:45.026+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:45.723 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:45 smithi119 podman[69539]: 2024-03-16 09:46:45.529287864 +0000 UTC m=+5.920569997 container cleanup 5f1ef8507d74cd722fb4b04edc36da0794a1a53f3194816c58c0787f06e6c636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-16T09:46:45.724 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:45 smithi119 bash[69539]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4 2024-03-16T09:46:45.991 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:45.622+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:45.991 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:45.622+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:46.119 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:45 smithi119 podman[69567]: 2024-03-16 09:46:45.722201176 +0000 UTC m=+0.600238338 container remove 5f1ef8507d74cd722fb4b04edc36da0794a1a53f3194816c58c0787f06e6c636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T09:46:46.120 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:45 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:45.798+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:46.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:45.989+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:46.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:45.989+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:46.377 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:46 smithi119 podman[69657]: 2024-03-16 09:46:46.040897437 +0000 UTC m=+0.017723194 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:46:46.377 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:46 smithi119 podman[69657]: 2024-03-16 09:46:46.13966968 +0000 UTC m=+0.116495469 container create ce74555754390729cdea2ce9ed02844f2aea096c6d3e5affc420f9045715cf8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-16T09:46:46.378 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:46 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:46.118+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:46.378 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:46 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:46.118+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:46.465 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:46 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:46.130+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:46.465 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:46 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:46.130+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:46.465 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:46 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:46.227+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:46.465 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:46 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:46.227+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:46.465 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:46 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:46.329+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:46.465 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:46 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:46.329+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:46.670 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:46 smithi119 podman[69657]: 2024-03-16 09:46:46.375820136 +0000 UTC m=+0.352645880 container init ce74555754390729cdea2ce9ed02844f2aea096c6d3e5affc420f9045715cf8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:46:46.671 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:46 smithi119 podman[69657]: 2024-03-16 09:46:46.379011253 +0000 UTC m=+0.355837002 container start ce74555754390729cdea2ce9ed02844f2aea096c6d3e5affc420f9045715cf8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:46:46.671 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:46 smithi119 podman[69657]: 2024-03-16 09:46:46.459041117 +0000 UTC m=+0.435866866 container attach ce74555754390729cdea2ce9ed02844f2aea096c6d3e5affc420f9045715cf8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T09:46:46.940 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:46 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:46.784+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:46.940 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 16 09:46:46 smithi119 podman[69657]: 2024-03-16 09:46:46.66948348 +0000 UTC m=+0.646309223 container died ce74555754390729cdea2ce9ed02844f2aea096c6d3e5affc420f9045715cf8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T09:46:47.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:46 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:46.638+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:47.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:46 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:46.638+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:47.280 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:47 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:47.103+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:47.281 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:47 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:47.103+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:47.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.4.service' 2024-03-16T09:46:47.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:47.382 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-16T09:46:47.382 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-16T09:46:47.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.5 2024-03-16T09:46:47.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:47 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:47.036+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:47.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:47 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:47.036+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:47.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:47.321+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:47.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:47.321+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:47.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:47.267+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:47.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:47.267+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:47.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:47.173+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:47.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:47.173+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:47.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:47 smithi129 systemd[1]: Stopping Ceph osd.5 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:48.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:47 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:47.677+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:48.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:47 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:47.677+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:48.051 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:47.706+0000 7f18be3bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:48.051 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:47.706+0000 7f18be3bb700 -1 osd.5 91 *** Got signal Terminated *** 2024-03-16T09:46:48.051 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:47 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:47.706+0000 7f18be3bb700 -1 osd.5 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:46:48.149 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:47 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:47.749+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:48.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:48 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:48.148+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:48.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:48 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:48.148+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:48.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:48 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:48.007+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:48.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:48 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:48.007+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:48.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:48.163+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:48.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:48.163+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:48.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:48.314+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:48.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:48.314+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:48.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:48.261+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:48.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:48 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:48.261+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:49.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:48 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:48.723+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:49.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:48 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:48.723+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:49.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:48 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:48.967+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:49.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:48 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:48.967+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:49.169 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:48 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:48.777+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:49.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:49 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:49.167+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:49.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:49 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:49.167+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:49.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:49 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:49.270+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:49.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:49 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:49.270+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:49.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:49 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:49.149+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:49.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:49 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:49.149+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:49.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:49 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:49.284+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:49.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:49 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:49.284+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:50.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:49 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:49.947+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:50.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:49 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:49.947+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:50.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:49 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:49.754+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:50.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:49 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:49.754+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:50.133 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:49 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:49.825+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:50.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:50 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:50.132+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:50.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:50 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:50.132+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:50.550 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:50 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:50.241+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:50.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:50 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:50.241+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:50.553 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:50 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:50.238+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:50.553 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:50 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:50.238+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:50.553 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:50 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:50.164+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:50.554 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:50 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:50.164+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:50.554 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:50 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:50.164+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:26.538408+0000 front 2024-03-16T09:46:26.538358+0000 (oldest deadline 2024-03-16T09:46:50.038525+0000) 2024-03-16T09:46:51.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:50.752+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:51.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:50.752+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:51.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:50.912+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:51.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:50.912+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:51.169 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:50 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:50.816+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:51.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:51 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:51.167+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:51.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:51 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:51.167+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:51.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:51 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:51.167+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:51.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:51 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:51.246+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:51.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:51 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:51.246+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:51.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:51 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:51.209+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:51.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:51 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:51.209+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:51.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:51 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:51.128+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:51.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:51 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:51.128+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:51.552 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:51 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:51.128+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:26.538408+0000 front 2024-03-16T09:46:26.538358+0000 (oldest deadline 2024-03-16T09:46:50.038525+0000) 2024-03-16T09:46:52.186 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:51 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:51.832+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:52.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:51 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:51.931+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:52.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:51 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:51.931+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:52.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:51 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:51.802+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:52.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:51 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:51.802+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:52.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:51 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:51.802+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:52.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:52 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:52.183+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:52.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:52 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:52.183+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:52.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:52 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:52.183+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:52.550 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:52.125+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:11.936171+0000 front 2024-03-16T09:46:11.936114+0000 (oldest deadline 2024-03-16T09:46:33.636142+0000) 2024-03-16T09:46:52.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:52.125+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.937171+0000 front 2024-03-16T09:46:18.937240+0000 (oldest deadline 2024-03-16T09:46:42.437215+0000) 2024-03-16T09:46:52.551 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5[47669]: 2024-03-16T09:46:52.125+0000 7f18b69a7700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:26.538408+0000 front 2024-03-16T09:46:26.538358+0000 (oldest deadline 2024-03-16T09:46:50.038525+0000) 2024-03-16T09:46:52.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:52.274+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:52.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:52.274+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:52.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:52.274+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:52.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:52.224+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:52.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:52.224+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:52.552 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:52 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:52.224+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:53.189 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:52 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:52.831+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:53.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:52.918+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:53.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:52.918+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:53.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:52.820+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:53.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:52.820+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:53.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:52 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:52.820+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:53.384 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:53 smithi129 podman[70368]: 2024-03-16 09:46:53.117970183 +0000 UTC m=+5.539964185 container died f3fabb2baa13fb8c35b7700697428a133ceb12a968b248003d9821e6adde5c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T09:46:53.385 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:53 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:53.280+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:53.385 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:53 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:53.280+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:53.385 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:53 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:53.280+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:53.385 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:53 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:53.269+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:53.385 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:53 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:53.269+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:53.385 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:53 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:53.269+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:53.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:53 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:53.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:53.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:53 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:53.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:53.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:53 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:53.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:53.752 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:53 smithi129 podman[70368]: 2024-03-16 09:46:53.490193015 +0000 UTC m=+5.912187016 container cleanup f3fabb2baa13fb8c35b7700697428a133ceb12a968b248003d9821e6adde5c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD) 2024-03-16T09:46:53.752 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:53 smithi129 bash[70368]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5 2024-03-16T09:46:54.050 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:53 smithi129 podman[70387]: 2024-03-16 09:46:53.750345869 +0000 UTC m=+0.626413383 container remove f3fabb2baa13fb8c35b7700697428a133ceb12a968b248003d9821e6adde5c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-16T09:46:54.189 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:53 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:53.865+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:54.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:53 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:53.865+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:46:54.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:53.808+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:54.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:53.808+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:54.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:53.808+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:54.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:53.891+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:54.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:53.891+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:54.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:53 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:53.891+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:46:54.364 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:54 smithi129 podman[70480]: 2024-03-16 09:46:54.002144145 +0000 UTC m=+0.018379136 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:46:54.364 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:54 smithi129 podman[70480]: 2024-03-16 09:46:54.109446684 +0000 UTC m=+0.125681663 container create 775d8fbee29f3503c2b6c0ffdefdd00e8085140c2e30ab192fda5fc446ff91e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T09:46:54.364 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:54 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:54.262+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:54.364 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:54 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:54.262+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:54.364 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:54 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:54.262+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:54.364 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:54 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:54.270+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:54.365 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:54 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:54.270+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:54.365 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:54 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:54.270+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:54.655 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:54 smithi129 podman[70480]: 2024-03-16 09:46:54.36192483 +0000 UTC m=+0.378159809 container init 775d8fbee29f3503c2b6c0ffdefdd00e8085140c2e30ab192fda5fc446ff91e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-16T09:46:54.655 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:54 smithi129 podman[70480]: 2024-03-16 09:46:54.365607402 +0000 UTC m=+0.381842386 container start 775d8fbee29f3503c2b6c0ffdefdd00e8085140c2e30ab192fda5fc446ff91e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:46:54.655 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:54 smithi129 podman[70480]: 2024-03-16 09:46:54.436765247 +0000 UTC m=+0.453000225 container attach 775d8fbee29f3503c2b6c0ffdefdd00e8085140c2e30ab192fda5fc446ff91e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T09:46:54.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:54 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:54.240+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:54.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:54 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:54.240+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:54.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:54 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:54.240+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:54.953 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 16 09:46:54 smithi129 podman[70480]: 2024-03-16 09:46:54.653620879 +0000 UTC m=+0.669855863 container died 775d8fbee29f3503c2b6c0ffdefdd00e8085140c2e30ab192fda5fc446ff91e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T09:46:55.189 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:54 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:54.876+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:55.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:54 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:54.876+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:46:55.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.5.service' 2024-03-16T09:46:55.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:54.835+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:55.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:54.835+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:55.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:54.835+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:55.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:54.877+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:55.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:54.877+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:55.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:54.877+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:46:55.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:46:55.271 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-16T09:46:55.271 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-16T09:46:55.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.6 2024-03-16T09:46:55.550 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:55 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:55.264+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:55.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:55 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:55.264+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:55.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:55 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:55.264+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:55.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:55 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:55.248+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:55.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:55 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:55.248+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:55.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:55 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:55.248+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:55.573 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:55 smithi077 systemd[1]: Stopping Ceph osd.6 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:46:55.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:55 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:55.270+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:55.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:55 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:55.270+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:55.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:55 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:55.270+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:55.842 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:55.570+0000 7f8fcef22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:46:55.842 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:55.570+0000 7f8fcef22700 -1 osd.6 91 *** Got signal Terminated *** 2024-03-16T09:46:55.842 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:55.570+0000 7f8fcef22700 -1 osd.6 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:46:56.189 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:55 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:55.843+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:56.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:55 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:55.843+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:46:56.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:55.880+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:56.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:55.880+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:56.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:55.880+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:56.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:55.840+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:56.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:55.840+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:56.260 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:55 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:55.840+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:46:56.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:56 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:56.255+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:56.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:56 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:56.255+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:56.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:56 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:56.255+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:56.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:56 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:56.288+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:56.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:56 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:56.288+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:56.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:56 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:56.288+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:56.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:56 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:56.261+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:56.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:56 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:56.261+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:56.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:56 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:56.261+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:57.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:56 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:56.818+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:57.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:56 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:56.818+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:46:57.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:56.871+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:57.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:56.871+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:57.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:56.871+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:57.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:56.860+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:57.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:56.860+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:57.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:56 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:56.860+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:46:57.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:57.243+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:57.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:57.243+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:57.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:57.243+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:57.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:57.222+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:57.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:57.222+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:57.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:57 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:57.222+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:57.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:57.291+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:57.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:57.291+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:57.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:57.291+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:58.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:57.790+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:58.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:57.790+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:46:58.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:57 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:57.790+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:46:58.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:57.856+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:58.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:57.856+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:58.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:57.856+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:58.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:57.908+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:58.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:57.908+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:58.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:57 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:57.908+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:46:58.550 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:58 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:58.245+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:58.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:58 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:58.245+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:58.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:58 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:58.245+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:58.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:58 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:58.247+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:58.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:58 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:58.247+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:58.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:58 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:58.247+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:58.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:58 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:58.316+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:58.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:58 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:58.316+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:58.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:58 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:58.316+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:46:59.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:58 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:58.760+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:46:59.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:58 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:58.760+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:46:59.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:58 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:58.760+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:46:59.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:58 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:58.849+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:46:59.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:58 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:58.849+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:46:59.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:58 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:58.849+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:46:59.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:58 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:58.849+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.793601+0000 front 2024-03-16T09:46:35.793532+0000 (oldest deadline 2024-03-16T09:46:58.093630+0000) 2024-03-16T09:46:59.260 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:58 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:58.944+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:46:59.260 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:58 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:58.944+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:46:59.260 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:58 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:58.944+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:46:59.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:59 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:59.260+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:46:59.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:59 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:59.260+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:46:59.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:46:59 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:46:59.260+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:46:59.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:59 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:59.196+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:46:59.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:59 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:59.196+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:46:59.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:59 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:59.196+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:46:59.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:46:59 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:46:59.196+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:46:59.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:59.333+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:46:59.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:59.333+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:46:59.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:46:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:46:59.333+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:00.189 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:59.711+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:00.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:59.711+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:00.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:46:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:46:59.711+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:00.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:59 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:59.912+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:00.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:59 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:59.912+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:00.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:46:59 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:46:59.912+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:00.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:59 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:59.879+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:09.688753+0000 front 2024-03-16T09:46:09.688826+0000 (oldest deadline 2024-03-16T09:46:34.388895+0000) 2024-03-16T09:47:00.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:59 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:59.879+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.990366+0000 front 2024-03-16T09:46:18.990348+0000 (oldest deadline 2024-03-16T09:46:41.890537+0000) 2024-03-16T09:47:00.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:59 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:59.879+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.092260+0000 front 2024-03-16T09:46:27.092286+0000 (oldest deadline 2024-03-16T09:46:51.092550+0000) 2024-03-16T09:47:00.259 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:46:59 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6[60761]: 2024-03-16T09:46:59.879+0000 7f8fcb516700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.793601+0000 front 2024-03-16T09:46:35.793532+0000 (oldest deadline 2024-03-16T09:46:58.093630+0000) 2024-03-16T09:47:00.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:00 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:00.220+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:00.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:00 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:00.220+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:00.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:00 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:00.220+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:00.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:00 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:00.225+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:00.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:00 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:00.225+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:00.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:00 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:00.226+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:00.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:00 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:00.226+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:00.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:00.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:00.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:00.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:00.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:00.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:00.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:00.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:01.189 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:00.704+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:01.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:00.704+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:01.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:00 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:00.704+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:01.232 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:00 smithi077 podman[92218]: 2024-03-16 09:47:00.96252788 +0000 UTC m=+5.491398533 container died 95c568056b944ab8f8f065ec69e7b006290ac1927031bf78bec7fcdadff50e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-16T09:47:01.233 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:00.938+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:01.233 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:00.938+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:01.233 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:00.938+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:01.233 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:00 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:00.938+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:01.509 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:01 smithi077 podman[92218]: 2024-03-16 09:47:01.430194753 +0000 UTC m=+5.959065461 container cleanup 95c568056b944ab8f8f065ec69e7b006290ac1927031bf78bec7fcdadff50e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:47:01.509 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:01 smithi077 bash[92218]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6 2024-03-16T09:47:01.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:01 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:01.213+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:01.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:01 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:01.213+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:01.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:01 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:01.213+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:01.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:01 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:01.263+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:01.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:01 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:01.263+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:01.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:01 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:01.263+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:01.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:01 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:01.263+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:01.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:01.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:01.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:01.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:01.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:01.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:01.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:01.297+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:02.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:01.734+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:02.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:01.734+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:02.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:01 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:01.734+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:02.192 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:01 smithi077 podman[92236]: 2024-03-16 09:47:01.798178741 +0000 UTC m=+0.829686284 container remove 95c568056b944ab8f8f065ec69e7b006290ac1927031bf78bec7fcdadff50e42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-16T09:47:02.192 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:01 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:01.894+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:02.192 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:01 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:01.894+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:02.192 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:01 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:01.894+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:02.192 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:01 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:01.894+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:02.462 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:02 smithi077 podman[92325]: 2024-03-16 09:47:02.094282583 +0000 UTC m=+0.018525932 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:47:02.462 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:02 smithi077 podman[92325]: 2024-03-16 09:47:02.19927547 +0000 UTC m=+0.123518806 container create 57cc27f47f552c34288c11825723301e04c3ffce0b503e50b9d88522631d7f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-16T09:47:02.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:02 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:02.202+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:02.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:02 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:02.202+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:02.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:02 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:02.202+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:02.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:02 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:02.265+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:02.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:02 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:02.265+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:02.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:02 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:02.265+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:02.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:02 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:02.265+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:02.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:02.298+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:02.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:02.298+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:02.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:02.298+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:02.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:02.298+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:02.756 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:02 smithi077 podman[92325]: 2024-03-16 09:47:02.460378712 +0000 UTC m=+0.384622049 container init 57cc27f47f552c34288c11825723301e04c3ffce0b503e50b9d88522631d7f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T09:47:02.756 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:02 smithi077 podman[92325]: 2024-03-16 09:47:02.463502243 +0000 UTC m=+0.387745584 container start 57cc27f47f552c34288c11825723301e04c3ffce0b503e50b9d88522631d7f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T09:47:02.756 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:02 smithi077 podman[92325]: 2024-03-16 09:47:02.535289655 +0000 UTC m=+0.459533008 container attach 57cc27f47f552c34288c11825723301e04c3ffce0b503e50b9d88522631d7f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:47:03.009 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 09:47:02 smithi077 podman[92325]: 2024-03-16 09:47:02.75421786 +0000 UTC m=+0.678461203 container died 57cc27f47f552c34288c11825723301e04c3ffce0b503e50b9d88522631d7f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T09:47:03.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:02 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:02.937+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:03.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:02 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:02.937+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:03.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:02 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:02.937+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:03.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:02 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:02.937+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:03.189 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:02.767+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:03.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:02.767+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:03.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:02.767+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:03.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.6.service' 2024-03-16T09:47:03.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:03.351 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-16T09:47:03.351 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-16T09:47:03.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.7 2024-03-16T09:47:03.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:03.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:03.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:03.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:03.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:03.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:03.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:03.266+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:03.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:03.266+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:03.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:03.266+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:03.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:03 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:03.266+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:03.683 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:03 smithi119 systemd[1]: Stopping Ceph osd.7 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:03.684 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:03.680+0000 7fed26826700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:03.684 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:03.680+0000 7fed26826700 -1 osd.7 91 *** Got signal Terminated *** 2024-03-16T09:47:03.684 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:03.680+0000 7fed26826700 -1 osd.7 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:47:03.684 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:03.334+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:03.684 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:03.334+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:03.684 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:03.334+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:03.684 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:03.334+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:03.940 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:03.809+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:03.940 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:03.809+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:03.940 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:03 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:03.809+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:04.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:03 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:03.911+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:04.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:03 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:03.911+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:04.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:03 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:03.911+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:04.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:03 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:03.911+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:04.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:04 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:04.165+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:04.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:04 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:04.165+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:04.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:04 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:04.165+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:04.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:04 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:04.315+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:04.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:04 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:04.315+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:04.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:04 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:04.315+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:04.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:04 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:04.315+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:04.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:04 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:04.350+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:04.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:04 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:04.350+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:04.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:04 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:04.350+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:04.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:04 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:04.350+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:05.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:04 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:04.803+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:05.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:04 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:04.803+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:05.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:04 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:04.803+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:05.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:04 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:04.878+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:05.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:04 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:04.878+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:05.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:04 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:04.878+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:05.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:04 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:04.878+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:05.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:05.192+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:05.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:05.192+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:05.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:05.192+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:05.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:05.347+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:05.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:05.347+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:05.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:05.347+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:05.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:05 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:05.347+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:05.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:05.367+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:05.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:05.367+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:05.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:05.367+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:05.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:05.367+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:05.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:05.367+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:06.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:05.769+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:06.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:05.769+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:06.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:05 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:05.769+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:06.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:05 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:05.835+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:06.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:05 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:05.835+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:06.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:05 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:05.835+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:06.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:05 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:05.835+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:06.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:05 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:05.835+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:06.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:06 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:06.150+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:06.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:06 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:06.150+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:06.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:06 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:06.150+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:06.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:06 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:06.364+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:06.552 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:06 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:06.364+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:06.552 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:06 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:06.364+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:06.552 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:06 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:06.364+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:06.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:06.319+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:06.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:06.319+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:06.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:06.319+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:06.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:06.319+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:06.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:06.319+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:07.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:06.763+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:07.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:06.763+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:07.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:06.763+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:07.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:06 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:06.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:07.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:06 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:06.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:07.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:06 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:06.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:07.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:06 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:06.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:07.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:06 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:06.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:07.163+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:07.163+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:07.163+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:07.163+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:07.357+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:07.357+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:07.357+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:07.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:07 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:07.357+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:07.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:07.294+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:07.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:07.294+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:07.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:07.294+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:07.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:07.294+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:07.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:07.294+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:08.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:07.735+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.092769+0000 front 2024-03-16T09:46:17.092833+0000 (oldest deadline 2024-03-16T09:46:42.992901+0000) 2024-03-16T09:47:08.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:07.735+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.093801+0000 front 2024-03-16T09:46:27.093768+0000 (oldest deadline 2024-03-16T09:46:52.993853+0000) 2024-03-16T09:47:08.190 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:07 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7[53601]: 2024-03-16T09:47:07.735+0000 7fed22e1a700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.494727+0000 front 2024-03-16T09:46:36.494710+0000 (oldest deadline 2024-03-16T09:46:57.594704+0000) 2024-03-16T09:47:08.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:07.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:08.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:07.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:08.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:07.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:08.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:07.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:08.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:07 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:07.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:08.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:08.368+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:08.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:08.368+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:08.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:08.368+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:08.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:08.368+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:08.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:08.179+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:08.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:08.179+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:08.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:08.179+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:08.553 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:08 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:08.179+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:08.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:08 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:08.258+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:08.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:08 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:08.258+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:08.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:08 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:08.258+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:08.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:08 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:08.258+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:08.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:08 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:08.258+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:09.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:08 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:08.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:09.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:08 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:08.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:09.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:08 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:08.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:09.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:08 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:08.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:09.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:08 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:08.848+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:09.325 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:09 smithi119 podman[69763]: 2024-03-16 09:47:09.046634365 +0000 UTC m=+5.507064619 container died 54ef4f83b909c8740864fcff7f781aa27c1f3569d2c4525d7b1c870c250780e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:47:09.326 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:09 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:09.224+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:09.326 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:09 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:09.224+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:09.326 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:09 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:09.224+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:09.326 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:09 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:09.224+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:09.326 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:09 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:09.224+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:09.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:09.351+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:09.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:09.351+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:09.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:09.351+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:09.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:09.351+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:09.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:09.228+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:09.552 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:09.228+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:09.552 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:09.228+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:09.552 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:09 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:09.228+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:09.675 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:09 smithi119 podman[69763]: 2024-03-16 09:47:09.431400666 +0000 UTC m=+5.891830926 container cleanup 54ef4f83b909c8740864fcff7f781aa27c1f3569d2c4525d7b1c870c250780e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux ) 2024-03-16T09:47:09.676 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:09 smithi119 bash[69763]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7 2024-03-16T09:47:09.939 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:09 smithi119 podman[69780]: 2024-03-16 09:47:09.674266375 +0000 UTC m=+0.631108412 container remove 54ef4f83b909c8740864fcff7f781aa27c1f3569d2c4525d7b1c870c250780e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:47:10.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:09 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:09.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:10.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:09 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:09.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:10.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:09 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:09.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:10.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:09 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:09.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:10.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:09 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:09.815+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:10.304 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:10 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:10.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:10.304 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:10 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:10.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:10.304 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:10 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:10.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:10.304 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:10 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:10.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:10.304 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:10 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:10.209+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:10.305 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:09 smithi119 podman[69870]: 2024-03-16 09:47:09.983433744 +0000 UTC m=+0.109629955 container create cd436b3211572486a983cbdcbdc0c9aa0f09f2c618766ea9219d7a002591b213 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-16T09:47:10.305 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:09 smithi119 podman[69870]: 2024-03-16 09:47:09.89258832 +0000 UTC m=+0.018784545 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:47:10.305 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:10 smithi119 podman[69870]: 2024-03-16 09:47:10.202440452 +0000 UTC m=+0.328636668 container init cd436b3211572486a983cbdcbdc0c9aa0f09f2c618766ea9219d7a002591b213 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:47:10.305 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:10 smithi119 podman[69870]: 2024-03-16 09:47:10.205425722 +0000 UTC m=+0.331621934 container start cd436b3211572486a983cbdcbdc0c9aa0f09f2c618766ea9219d7a002591b213 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T09:47:10.305 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:10 smithi119 podman[69870]: 2024-03-16 09:47:10.30249473 +0000 UTC m=+0.428690946 container attach cd436b3211572486a983cbdcbdc0c9aa0f09f2c618766ea9219d7a002591b213 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T09:47:10.550 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:10.396+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:10.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:10.396+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:10.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:10.396+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:10.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:10.396+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:10.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:10.276+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:10.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:10.276+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:10.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:10.276+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:10.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:10 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:10.276+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:10.690 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 16 09:47:10 smithi119 podman[69870]: 2024-03-16 09:47:10.497719847 +0000 UTC m=+0.623916059 container died cd436b3211572486a983cbdcbdc0c9aa0f09f2c618766ea9219d7a002591b213 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:47:11.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.7.service' 2024-03-16T09:47:11.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:11.111 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-16T09:47:11.111 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-16T09:47:11.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.8 2024-03-16T09:47:11.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:10 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:10.797+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:11.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:10 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:10.797+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:11.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:10 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:10.797+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:11.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:10 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:10.797+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:11.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:10 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:10.797+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:11.323 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:11.160+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:11.323 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:11.160+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:11.323 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:11.160+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:11.323 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:11.160+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:11.323 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:11 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:11.160+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:11.405 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 systemd[1]: Stopping Ceph osd.8 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:11.405 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:11.325+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:11.405 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:11.325+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:11.405 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:11.325+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:11.405 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:11.325+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:11.800 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:11.402+0000 7f000530a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:11.801 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:11.402+0000 7f000530a700 -1 osd.8 91 *** Got signal Terminated *** 2024-03-16T09:47:11.801 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:11.402+0000 7f000530a700 -1 osd.8 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:47:11.801 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:11.408+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:11.801 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:11.408+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:11.801 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:11.408+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:11.801 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:11 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:11.408+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:12.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:11.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:12.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:11.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:12.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:11.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:12.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:11.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:12.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:11 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:11.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:12.377+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:12.377+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:12.377+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:12.377+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:12.285+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:12.285+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:12.285+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:12.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:12.285+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:12.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:12.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:12.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:12.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:12.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:12.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:12.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:12.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:12.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:12 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:12.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:13.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:12.727+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:13.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:12.727+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:13.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:12.727+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:13.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:12.727+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:13.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:12 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:12.727+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:13.550 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:13.349+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:13.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:13.349+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:13.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:13.349+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:13.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:13.349+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:13.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:13.265+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:13.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:13.265+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:13.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:13.265+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:13.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:13 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:13.265+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:13.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:13 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:13.234+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:13.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:13 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:13.234+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:13.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:13 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:13.234+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:13.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:13 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:13.234+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:13.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:13 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:13.234+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:14.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:13 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:13.751+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:14.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:13 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:13.751+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:14.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:13 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:13.751+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:14.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:13 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:13.751+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:14.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:13 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:13.751+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:14.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:14.261+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:14.261+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:14.261+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:14.261+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:14.386+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:14.386+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:14.386+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:14.386+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:14.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:14 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:14.386+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:50.624675+0000 front 2024-03-16T09:46:50.624720+0000 (oldest deadline 2024-03-16T09:47:14.124768+0000) 2024-03-16T09:47:14.591 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:14.227+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:14.591 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:14.227+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:14.591 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:14.227+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:14.591 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:14.227+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:14.591 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:14.227+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:14.591 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:14 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:14.227+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:15.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:14.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:15.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:14.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:15.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:14.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:15.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:14.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:15.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:14 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:14.757+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:15.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:15.236+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:15.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:15.236+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:15.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:15.236+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:15.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:15.236+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:15.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:15.425+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:15.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:15.425+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:15.552 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:15.425+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:15.552 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:15.425+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:15.552 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:15 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:15.425+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:50.624675+0000 front 2024-03-16T09:46:50.624720+0000 (oldest deadline 2024-03-16T09:47:14.124768+0000) 2024-03-16T09:47:15.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:15.272+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:15.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:15.272+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:15.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:15.272+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:15.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:15.272+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:15.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:15.272+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:15.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:15 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:15.272+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:16.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:15 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:15.788+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:16.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:15 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:15.788+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:16.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:15 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:15.788+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:16.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:15 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:15.788+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:16.260 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:15 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:15.788+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:16.375+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:10.617394+0000 front 2024-03-16T09:46:10.617370+0000 (oldest deadline 2024-03-16T09:46:33.517403+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:16.375+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.018596+0000 front 2024-03-16T09:46:17.018617+0000 (oldest deadline 2024-03-16T09:46:42.918668+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:16.375+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:25.720121+0000 front 2024-03-16T09:46:25.720148+0000 (oldest deadline 2024-03-16T09:46:51.620354+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:16.375+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:31.621376+0000 front 2024-03-16T09:46:31.621311+0000 (oldest deadline 2024-03-16T09:46:58.621355+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8[54522]: 2024-03-16T09:47:16.375+0000 7f00018fe700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:50.624675+0000 front 2024-03-16T09:46:50.624720+0000 (oldest deadline 2024-03-16T09:47:14.124768+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:16.199+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:16.199+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:16.199+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:16.199+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:16.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:16 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:16.199+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:16.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:16 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:16.230+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:16.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:16 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:16.230+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:16.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:16 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:16.230+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:16.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:16 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:16.230+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:16.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:16 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:16.230+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:16.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:16 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:16.230+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:17.048 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:16 smithi129 podman[70584]: 2024-03-16 09:47:16.73840856 +0000 UTC m=+5.430860658 container died b53bbfde2470c925de6eb3cc5cf5bf73c2277e192ec07d13730f839b91248c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-16T09:47:17.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:16.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:17.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:16.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:17.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:16.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:17.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:16.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:17.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:16 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:16.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:17.300 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:17 smithi129 podman[70584]: 2024-03-16 09:47:17.144373216 +0000 UTC m=+5.836825322 container cleanup b53bbfde2470c925de6eb3cc5cf5bf73c2277e192ec07d13730f839b91248c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-16T09:47:17.301 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:17 smithi129 bash[70584]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8 2024-03-16T09:47:17.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:17 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:17.169+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:17.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:17 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:17.169+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:17.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:17 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:17.169+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:17.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:17 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:17.169+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:17.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:17 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:17.169+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:17.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:17.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:17.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:17.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:17.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:17.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:17.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:17.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:17.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:17.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:17.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:17 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:17.207+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:17.761 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:17 smithi129 podman[70615]: 2024-03-16 09:47:17.462839011 +0000 UTC m=+0.729060670 container remove b53bbfde2470c925de6eb3cc5cf5bf73c2277e192ec07d13730f839b91248c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T09:47:18.042 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:17 smithi129 podman[70705]: 2024-03-16 09:47:17.662496037 +0000 UTC m=+0.017845931 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:47:18.042 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:17 smithi129 podman[70705]: 2024-03-16 09:47:17.763392121 +0000 UTC m=+0.118742008 container create 16d470622754d94e75f029941ef0bbcbbd5df35868b9137d7ad208d37a8b657d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:47:18.042 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:17 smithi129 podman[70705]: 2024-03-16 09:47:17.966278391 +0000 UTC m=+0.321628287 container init 16d470622754d94e75f029941ef0bbcbbd5df35868b9137d7ad208d37a8b657d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-16T09:47:18.042 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:17 smithi129 podman[70705]: 2024-03-16 09:47:17.969507872 +0000 UTC m=+0.324857760 container start 16d470622754d94e75f029941ef0bbcbbd5df35868b9137d7ad208d37a8b657d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:47:18.043 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:18 smithi129 podman[70705]: 2024-03-16 09:47:18.040603076 +0000 UTC m=+0.395952975 container attach 16d470622754d94e75f029941ef0bbcbbd5df35868b9137d7ad208d37a8b657d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T09:47:18.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:17 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:17.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:18.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:17 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:17.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:18.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:17 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:17.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:18.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:17 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:17.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:18.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:17 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:17.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:18.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:17 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:17.782+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:51.234730+0000 front 2024-03-16T09:46:51.234655+0000 (oldest deadline 2024-03-16T09:47:17.134634+0000) 2024-03-16T09:47:18.300 INFO:journalctl@ceph.osd.8.smithi129.stdout:Mar 16 09:47:18 smithi129 podman[70705]: 2024-03-16 09:47:18.268234221 +0000 UTC m=+0.623584110 container died 16d470622754d94e75f029941ef0bbcbbd5df35868b9137d7ad208d37a8b657d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T09:47:18.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:18 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:18.197+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:18.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:18 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:18.197+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:18.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:18 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:18.197+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:18.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:18 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:18.197+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:18.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:18 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:18.197+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:18.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:18 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:18.201+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:18.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:18 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:18.201+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:18.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:18 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:18.201+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:18.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:18 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:18.201+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:18.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:18 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:18.201+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:18.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:18 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:18.201+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:18.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.8.service' 2024-03-16T09:47:18.880 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:18.880 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-16T09:47:18.880 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-16T09:47:18.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.9 2024-03-16T09:47:19.185 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:18 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:18.779+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:19.185 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:18 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:18.779+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:19.186 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:18 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:18.779+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:19.186 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:18 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:18.779+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:19.186 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:18 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:18.779+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:19.186 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:18 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:18.779+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:51.234730+0000 front 2024-03-16T09:46:51.234655+0000 (oldest deadline 2024-03-16T09:47:17.134634+0000) 2024-03-16T09:47:19.186 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:18 smithi077 systemd[1]: Stopping Ceph osd.9 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:19.508 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.182+0000 7f6048e06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:19.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.182+0000 7f6048e06700 -1 osd.9 91 *** Got signal Terminated *** 2024-03-16T09:47:19.509 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.182+0000 7f6048e06700 -1 osd.9 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:47:19.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:19 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:19.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:19.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:19 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:19.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:19.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:19 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:19.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:19.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:19 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:19.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:19.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:19 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:19.172+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:19.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:19 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:19.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:19.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:19 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:19.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:19.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:19 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:19.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:19.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:19 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:19.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:19.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:19 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:19.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:19.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:19 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:19.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:20.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.767+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:20.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.767+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:20.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.767+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:20.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.767+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:20.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.767+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:20.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:19 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:19.767+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:51.234730+0000 front 2024-03-16T09:46:51.234655+0000 (oldest deadline 2024-03-16T09:47:17.134634+0000) 2024-03-16T09:47:20.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:20 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:20.147+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:20.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:20 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:20.147+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:20.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:20 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:20.147+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:20.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:20 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:20.147+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:20.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:20 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:20.147+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:20.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:20 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:20.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:20.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:20 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:20.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:20.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:20 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:20.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:20.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:20 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:20.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:20.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:20 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:20.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:20.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:20 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:20.200+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:21.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:20 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:20.745+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:21.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:20 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:20.745+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:21.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:20 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:20.745+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:21.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:20 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:20.745+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:21.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:20 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:20.745+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:21.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:20 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:20.745+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:51.234730+0000 front 2024-03-16T09:46:51.234655+0000 (oldest deadline 2024-03-16T09:47:17.134634+0000) 2024-03-16T09:47:21.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:21 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:21.133+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:21.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:21 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:21.133+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:21.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:21 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:21.133+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:21.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:21 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:21.133+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:21.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:21 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:21.133+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:21.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:21 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:21.218+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:21.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:21 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:21.218+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:21.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:21 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:21.218+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:21.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:21 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:21.218+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:21.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:21 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:21.218+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:21.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:21 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:21.218+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:22.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:21 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:21.746+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:22.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:21 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:21.746+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:22.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:21 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:21.746+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:22.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:21 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:21.746+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:22.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:21 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:21.746+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:22.009 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:21 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:21.746+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:51.234730+0000 front 2024-03-16T09:46:51.234655+0000 (oldest deadline 2024-03-16T09:47:17.134634+0000) 2024-03-16T09:47:22.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:22.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:22.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:22.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:22.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:22.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:22.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:22.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:22.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:22.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:22.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:22 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:22.215+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:22.689 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:22 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:22.215+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:22.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:22 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:22.215+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:22.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:22 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:22.215+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:22.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:22 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:22.215+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:22.690 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:22 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:22.215+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:23.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:22 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:22.791+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:23.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:22 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:22.791+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:23.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:22 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:22.791+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:23.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:22 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:22.791+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:23.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:22 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:22.791+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:23.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:22 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:22.791+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:51.234730+0000 front 2024-03-16T09:46:51.234655+0000 (oldest deadline 2024-03-16T09:47:17.134634+0000) 2024-03-16T09:47:23.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:23 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:23.165+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:23.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:23 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:23.165+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:23.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:23 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:23.165+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:23.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:23 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:23.165+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:23.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:23 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:23.165+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:23.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:23 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:23.165+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:23.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:23.113+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:23.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:23.113+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:23.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:23.113+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:23.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:23.113+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:23.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:23.113+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:23.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:23 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:23.113+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:24.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:23.825+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.530046+0000 front 2024-03-16T09:46:12.530021+0000 (oldest deadline 2024-03-16T09:46:34.830171+0000) 2024-03-16T09:47:24.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:23.825+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:18.331288+0000 front 2024-03-16T09:46:18.331385+0000 (oldest deadline 2024-03-16T09:46:41.231129+0000) 2024-03-16T09:47:24.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:23.825+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:27.132326+0000 front 2024-03-16T09:46:27.132360+0000 (oldest deadline 2024-03-16T09:46:53.032122+0000) 2024-03-16T09:47:24.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:23.825+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:36.533063+0000 front 2024-03-16T09:46:36.533125+0000 (oldest deadline 2024-03-16T09:47:00.033303+0000) 2024-03-16T09:47:24.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:23.825+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:40.033803+0000 front 2024-03-16T09:46:40.033820+0000 (oldest deadline 2024-03-16T09:47:05.333748+0000) 2024-03-16T09:47:24.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:23.825+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:51.234730+0000 front 2024-03-16T09:46:51.234655+0000 (oldest deadline 2024-03-16T09:47:17.134634+0000) 2024-03-16T09:47:24.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:23 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9[67780]: 2024-03-16T09:47:23.825+0000 7f60453fa700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:46:57.135089+0000 front 2024-03-16T09:46:57.135121+0000 (oldest deadline 2024-03-16T09:47:23.535581+0000) 2024-03-16T09:47:24.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:24.151+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:24.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:24.151+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:24.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:24.151+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:24.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:24.151+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:24.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:24.151+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:24.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:24 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:24.151+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:24.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:24 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:24.128+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:24.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:24 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:24.128+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:24.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:24 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:24.128+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:24.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:24 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:24.128+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:24.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:24 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:24.128+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:24.552 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:24 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:24.128+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:24.984 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:24 smithi077 podman[92443]: 2024-03-16 09:47:24.662625851 +0000 UTC m=+5.591588797 container died 4fca0808fd10a23dcdc7d328fc989c655287d9bab3497dc9f050e9dec52bbe03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-16T09:47:25.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:24 smithi077 podman[92443]: 2024-03-16 09:47:24.982724941 +0000 UTC m=+5.911687896 container cleanup 4fca0808fd10a23dcdc7d328fc989c655287d9bab3497dc9f050e9dec52bbe03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T09:47:25.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:24 smithi077 bash[92443]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9 2024-03-16T09:47:25.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:25 smithi077 podman[92460]: 2024-03-16 09:47:25.175459619 +0000 UTC m=+0.514909556 container remove 4fca0808fd10a23dcdc7d328fc989c655287d9bab3497dc9f050e9dec52bbe03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T09:47:25.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:25 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:25.133+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:25.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:25 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:25.133+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:25.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:25 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:25.133+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:25.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:25 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:25.133+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:25.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:25 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:25.133+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:25.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:25 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:25.133+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:25.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:25 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:25.133+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.248219+0000 front 2024-03-16T09:47:00.248301+0000 (oldest deadline 2024-03-16T09:47:24.947990+0000) 2024-03-16T09:47:25.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:25 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:25.099+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:25.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:25 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:25.099+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:25.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:25 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:25.099+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:25.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:25 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:25.099+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:25.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:25 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:25.099+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:25.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:25 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:25.099+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:25.888 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:25 smithi077 podman[92552]: 2024-03-16 09:47:25.49067619 +0000 UTC m=+0.017699398 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:47:25.888 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:25 smithi077 podman[92552]: 2024-03-16 09:47:25.592930435 +0000 UTC m=+0.119953634 container create 9cc67aeaaf22f7f89a95df2ff41059ba6d1101f2e5e4b988c84e153c5638deb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-16T09:47:25.889 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:25 smithi077 podman[92552]: 2024-03-16 09:47:25.788680006 +0000 UTC m=+0.315703207 container init 9cc67aeaaf22f7f89a95df2ff41059ba6d1101f2e5e4b988c84e153c5638deb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T09:47:25.889 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:25 smithi077 podman[92552]: 2024-03-16 09:47:25.791913926 +0000 UTC m=+0.318937128 container start 9cc67aeaaf22f7f89a95df2ff41059ba6d1101f2e5e4b988c84e153c5638deb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:47:25.889 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:25 smithi077 podman[92552]: 2024-03-16 09:47:25.886823453 +0000 UTC m=+0.413846660 container attach 9cc67aeaaf22f7f89a95df2ff41059ba6d1101f2e5e4b988c84e153c5638deb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:47:26.259 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:26 smithi077 podman[92552]: 2024-03-16 09:47:26.077021553 +0000 UTC m=+0.604044762 container died 9cc67aeaaf22f7f89a95df2ff41059ba6d1101f2e5e4b988c84e153c5638deb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-16T09:47:26.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.134+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:26.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.134+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:26.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.134+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:26.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.134+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:26.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.134+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:26.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.134+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:26.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.134+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.248219+0000 front 2024-03-16T09:47:00.248301+0000 (oldest deadline 2024-03-16T09:47:24.947990+0000) 2024-03-16T09:47:26.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:26.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:26.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:26.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:26.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:26.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:26.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:26.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:26.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:26.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:26.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:26.137+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:26.610 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 09:47:26 smithi077 podman[92578]: 2024-03-16 09:47:26.590438311 +0000 UTC m=+0.508676979 container remove 9cc67aeaaf22f7f89a95df2ff41059ba6d1101f2e5e4b988c84e153c5638deb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T09:47:26.614 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.9.service' 2024-03-16T09:47:26.654 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:26.654 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-16T09:47:26.654 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-16T09:47:26.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.10 2024-03-16T09:47:26.939 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 systemd[1]: Stopping Ceph osd.10 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.965+0000 7ff66a8e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.965+0000 7ff66a8e0700 -1 osd.10 91 *** Got signal Terminated *** 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:26 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:26.965+0000 7ff66a8e0700 -1 osd.10 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:27 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:27.101+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:27 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:27.101+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:27 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:27.101+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:27 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:27.101+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:27 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:27.101+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:27 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:27.101+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:27.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:27 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:27.101+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.248219+0000 front 2024-03-16T09:47:00.248301+0000 (oldest deadline 2024-03-16T09:47:24.947990+0000) 2024-03-16T09:47:27.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:27 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:27.136+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:27.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:27 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:27.136+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:27.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:27 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:27.136+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:27.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:27 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:27.136+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:27.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:27 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:27.136+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:27.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:27 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:27.136+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:28.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:28 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:28.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:28.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:28 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:28.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:28.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:28 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:28.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:28.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:28 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:28.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:28.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:28 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:28.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:28.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:28 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:28.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:28.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:28 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:28.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.248219+0000 front 2024-03-16T09:47:00.248301+0000 (oldest deadline 2024-03-16T09:47:24.947990+0000) 2024-03-16T09:47:28.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:28 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:28.117+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:28.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:28 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:28.117+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:28.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:28 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:28.117+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:28.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:28 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:28.117+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:28.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:28 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:28.117+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:28.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:28 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:28.117+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:29.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:29 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:29.053+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:29.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:29 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:29.053+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:29.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:29 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:29.053+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:29.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:29 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:29.053+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:29.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:29 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:29.053+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:29.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:29 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:29.053+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:29.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:29 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:29.053+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.248219+0000 front 2024-03-16T09:47:00.248301+0000 (oldest deadline 2024-03-16T09:47:24.947990+0000) 2024-03-16T09:47:29.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:29 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:29.127+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:29.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:29 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:29.127+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:29.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:29 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:29.127+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:29.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:29 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:29.127+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:29.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:29 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:29.127+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:29.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:29 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:29.127+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:30.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:30 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:30.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:30.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:30 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:30.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:30.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:30 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:30.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:30.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:30 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:30.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:30.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:30 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:30.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:30.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:30 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:30.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:30.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:30 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:30.064+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.248219+0000 front 2024-03-16T09:47:00.248301+0000 (oldest deadline 2024-03-16T09:47:24.947990+0000) 2024-03-16T09:47:30.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:30 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:30.081+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:30.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:30 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:30.081+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:30.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:30 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:30.081+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:30.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:30 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:30.081+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:30.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:30 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:30.081+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:30.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:30 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:30.081+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:30.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:30 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:30.081+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:31.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:31 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:31.043+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:31.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:31 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:31.043+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:31.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:31 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:31.043+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:31.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:31 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:31.043+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:31.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:31 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:31.043+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:31.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:31 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:31.043+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:31.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:31 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:31.043+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:31.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:31.077+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:12.338448+0000 front 2024-03-16T09:46:12.338353+0000 (oldest deadline 2024-03-16T09:46:37.638267+0000) 2024-03-16T09:47:31.439 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:31.077+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:19.839777+0000 front 2024-03-16T09:46:19.839747+0000 (oldest deadline 2024-03-16T09:46:43.939941+0000) 2024-03-16T09:47:31.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:31.077+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:28.041110+0000 front 2024-03-16T09:46:28.040983+0000 (oldest deadline 2024-03-16T09:46:50.340772+0000) 2024-03-16T09:47:31.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:31.077+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T09:46:35.943582+0000 front 2024-03-16T09:46:35.943526+0000 (oldest deadline 2024-03-16T09:47:00.043449+0000) 2024-03-16T09:47:31.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:31.077+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:44.044794+0000 front 2024-03-16T09:46:44.044828+0000 (oldest deadline 2024-03-16T09:47:05.144932+0000) 2024-03-16T09:47:31.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:31.077+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:52.146587+0000 front 2024-03-16T09:46:52.146634+0000 (oldest deadline 2024-03-16T09:47:13.846525+0000) 2024-03-16T09:47:31.440 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:31 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10[60630]: 2024-03-16T09:47:31.077+0000 7ff666ed4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.248219+0000 front 2024-03-16T09:47:00.248301+0000 (oldest deadline 2024-03-16T09:47:24.947990+0000) 2024-03-16T09:47:32.550 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:32 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:32.067+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:32.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:32 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:32.067+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:32.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:32 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:32.067+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:32.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:32 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:32.067+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:32.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:32 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:32.067+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:32.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:32 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:32.067+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:32.551 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:32 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:32.067+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:32.601 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:32 smithi119 podman[69988]: 2024-03-16 09:47:32.339484177 +0000 UTC m=+5.492706966 container died 0da29803811c94be6a2ea91d3ff7ed913fbb320917971ae05c308ffe6ba7080f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-16T09:47:32.939 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:32 smithi119 podman[69988]: 2024-03-16 09:47:32.698628447 +0000 UTC m=+5.851851234 container cleanup 0da29803811c94be6a2ea91d3ff7ed913fbb320917971ae05c308ffe6ba7080f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T09:47:32.940 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:32 smithi119 bash[69988]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10 2024-03-16T09:47:33.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:33 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:33.032+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:33.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:33 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:33.032+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:33.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:33 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:33.032+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:33.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:33 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:33.032+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:33.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:33 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:33.032+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:33.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:33 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:33.032+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:33.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:33 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:33.032+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:33.308 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:32 smithi119 podman[70005]: 2024-03-16 09:47:32.976550002 +0000 UTC m=+0.631431711 container remove 0da29803811c94be6a2ea91d3ff7ed913fbb320917971ae05c308ffe6ba7080f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-16T09:47:33.615 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:33 smithi119 podman[70097]: 2024-03-16 09:47:33.209786929 +0000 UTC m=+0.017466312 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:47:33.615 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:33 smithi119 podman[70097]: 2024-03-16 09:47:33.352972298 +0000 UTC m=+0.160651680 container create b2fcb61dccd9a197b3cc4eba7c4ebf812655a4889de8f2a8edc9545bdb310ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T09:47:33.615 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:33 smithi119 podman[70097]: 2024-03-16 09:47:33.530480184 +0000 UTC m=+0.338159567 container init b2fcb61dccd9a197b3cc4eba7c4ebf812655a4889de8f2a8edc9545bdb310ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-16T09:47:33.616 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:33 smithi119 podman[70097]: 2024-03-16 09:47:33.534831061 +0000 UTC m=+0.342510447 container start b2fcb61dccd9a197b3cc4eba7c4ebf812655a4889de8f2a8edc9545bdb310ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T09:47:33.616 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:33 smithi119 podman[70097]: 2024-03-16 09:47:33.613551663 +0000 UTC m=+0.421231059 container attach b2fcb61dccd9a197b3cc4eba7c4ebf812655a4889de8f2a8edc9545bdb310ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-16T09:47:33.939 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 16 09:47:33 smithi119 podman[70097]: 2024-03-16 09:47:33.813608212 +0000 UTC m=+0.621287596 container died b2fcb61dccd9a197b3cc4eba7c4ebf812655a4889de8f2a8edc9545bdb310ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T09:47:34.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.006+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:34.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.006+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:34.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.006+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:34.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.006+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:34.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.006+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:34.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.006+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:34.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.006+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:34.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.10.service' 2024-03-16T09:47:34.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:34.426 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-16T09:47:34.426 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-16T09:47:34.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.11 2024-03-16T09:47:34.732 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 systemd[1]: Stopping Ceph osd.11 for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:35.000 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.729+0000 7fa28bdaf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:35.001 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.729+0000 7fa28bdaf700 -1 osd.11 91 *** Got signal Terminated *** 2024-03-16T09:47:35.001 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.729+0000 7fa28bdaf700 -1 osd.11 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T09:47:35.001 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.998+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:35.001 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.998+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:35.001 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.998+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:35.001 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.998+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:35.001 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.998+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:35.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.998+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:35.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:34 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:34.998+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:36.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:35.991+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:36.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:35.991+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:36.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:35.991+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:36.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:35.991+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:36.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:35.991+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:36.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:35.991+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:36.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:35 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:35.991+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:37.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:36.969+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:37.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:36.969+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:37.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:36.969+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:37.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:36.969+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:37.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:36.969+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:37.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:36.969+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:37.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:36 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:36.969+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:38.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:37.999+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:38.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:37.999+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:38.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:37.999+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:38.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:37.999+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:38.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:37.999+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:38.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:37.999+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:38.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:37.999+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:39.300 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:38.952+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T09:46:08.908898+0000 front 2024-03-16T09:46:08.909028+0000 (oldest deadline 2024-03-16T09:46:33.608892+0000) 2024-03-16T09:47:39.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:38.952+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-16T09:46:17.109945+0000 front 2024-03-16T09:46:17.109935+0000 (oldest deadline 2024-03-16T09:46:43.009961+0000) 2024-03-16T09:47:39.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:38.952+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6804 osd.2 since back 2024-03-16T09:46:23.010285+0000 front 2024-03-16T09:46:23.010316+0000 (oldest deadline 2024-03-16T09:46:51.811126+0000) 2024-03-16T09:47:39.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:38.952+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-16T09:46:43.413720+0000 front 2024-03-16T09:46:43.413629+0000 (oldest deadline 2024-03-16T09:47:06.313532+0000) 2024-03-16T09:47:39.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:38.952+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.129:6812 osd.5 since back 2024-03-16T09:46:49.714883+0000 front 2024-03-16T09:46:49.714772+0000 (oldest deadline 2024-03-16T09:47:15.614984+0000) 2024-03-16T09:47:39.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:38.952+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T09:47:00.315932+0000 front 2024-03-16T09:47:00.315972+0000 (oldest deadline 2024-03-16T09:47:22.615922+0000) 2024-03-16T09:47:39.301 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:38 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11[61411]: 2024-03-16T09:47:38.952+0000 7fa28439b700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-16T09:47:08.516968+0000 front 2024-03-16T09:47:08.517013+0000 (oldest deadline 2024-03-16T09:47:29.616905+0000) 2024-03-16T09:47:40.399 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:40 smithi129 podman[70810]: 2024-03-16 09:47:40.098311399 +0000 UTC m=+5.483586598 container died 25916333ee4eeae1d7be28c8d73ed2d829394a17f5be7dd6bb8e630a38433ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-16T09:47:40.740 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:40 smithi129 podman[70810]: 2024-03-16 09:47:40.470986173 +0000 UTC m=+5.856261363 container cleanup 25916333ee4eeae1d7be28c8d73ed2d829394a17f5be7dd6bb8e630a38433ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T09:47:40.740 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:40 smithi129 bash[70810]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11 2024-03-16T09:47:41.050 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:40 smithi129 podman[70827]: 2024-03-16 09:47:40.7387242 +0000 UTC m=+0.634662238 container remove 25916333ee4eeae1d7be28c8d73ed2d829394a17f5be7dd6bb8e630a38433ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-16T09:47:41.402 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:41 smithi129 podman[70917]: 2024-03-16 09:47:40.987027694 +0000 UTC m=+0.018429937 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T09:47:41.402 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:41 smithi129 podman[70917]: 2024-03-16 09:47:41.106105732 +0000 UTC m=+0.137507974 container create d2d693e7876c09900b5a69471072f10f655a41177a115e8033e7f81be047c99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T09:47:41.402 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:41 smithi129 podman[70917]: 2024-03-16 09:47:41.308684474 +0000 UTC m=+0.340086725 container init d2d693e7876c09900b5a69471072f10f655a41177a115e8033e7f81be047c99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-16T09:47:41.402 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:41 smithi129 podman[70917]: 2024-03-16 09:47:41.312365769 +0000 UTC m=+0.343768013 container start d2d693e7876c09900b5a69471072f10f655a41177a115e8033e7f81be047c99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-16T09:47:41.402 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:41 smithi129 podman[70917]: 2024-03-16 09:47:41.400334884 +0000 UTC m=+0.431737141 container attach d2d693e7876c09900b5a69471072f10f655a41177a115e8033e7f81be047c99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:47:41.800 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:41 smithi129 podman[70917]: 2024-03-16 09:47:41.601073052 +0000 UTC m=+0.632475286 container died d2d693e7876c09900b5a69471072f10f655a41177a115e8033e7f81be047c99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T09:47:42.148 INFO:journalctl@ceph.osd.11.smithi129.stdout:Mar 16 09:47:42 smithi129 podman[70943]: 2024-03-16 09:47:42.145045364 +0000 UTC m=+0.535490017 container remove d2d693e7876c09900b5a69471072f10f655a41177a115e8033e7f81be047c99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T09:47:42.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@osd.11.service' 2024-03-16T09:47:42.212 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:42.213 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-16T09:47:42.213 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-16T09:47:42.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.a 2024-03-16T09:47:42.503 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 09:47:42 smithi077 systemd[1]: Stopping Ceph mds.a for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:42.758 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 09:47:42 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-a[74301]: 2024-03-16T09:47:42.500+0000 7f573fc88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:42.759 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 09:47:42 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-a[74301]: 2024-03-16T09:47:42.500+0000 7f573fc88700 -1 mds.a *** got signal Terminated *** 2024-03-16T09:47:44.259 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 09:47:43 smithi077 podman[92669]: 2024-03-16 09:47:43.971840391 +0000 UTC m=+1.561979044 container died 443f99b040ffd97617a398d623a207409df49213600b24f1194cd63c793b25d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-a, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-16T09:47:44.570 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 09:47:44 smithi077 podman[92669]: 2024-03-16 09:47:44.35911158 +0000 UTC m=+1.949250233 container cleanup 443f99b040ffd97617a398d623a207409df49213600b24f1194cd63c793b25d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T09:47:44.570 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 09:47:44 smithi077 bash[92669]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-a 2024-03-16T09:47:44.701 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.a.service' 2024-03-16T09:47:44.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:44.737 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-16T09:47:44.738 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-16T09:47:44.738 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.d 2024-03-16T09:47:45.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:47:44 smithi077 systemd[1]: Stopping Ceph mds.d for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:45.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:47:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d[73354]: 2024-03-16T09:47:45.070+0000 7fd9cdfa4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:45.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:47:45 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d[73354]: 2024-03-16T09:47:45.070+0000 7fd9cdfa4700 -1 mds.d *** got signal Terminated *** 2024-03-16T09:47:49.259 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:47:48 smithi077 podman[92791]: 2024-03-16 09:47:48.968789166 +0000 UTC m=+4.012563432 container died d6b5984fc6ae12664bb522de7d106b974d622f6cfca11301868dfde466c125d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-16T09:47:49.701 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:47:49 smithi077 podman[92791]: 2024-03-16 09:47:49.29622805 +0000 UTC m=+4.340002315 container cleanup d6b5984fc6ae12664bb522de7d106b974d622f6cfca11301868dfde466c125d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-16T09:47:49.701 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:47:49 smithi077 bash[92791]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d 2024-03-16T09:47:49.701 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 09:47:49 smithi077 podman[92805]: 2024-03-16 09:47:49.480754085 +0000 UTC m=+0.510170412 container remove d6b5984fc6ae12664bb522de7d106b974d622f6cfca11301868dfde466c125d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-16T09:47:49.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.d.service' 2024-03-16T09:47:49.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:49.746 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-16T09:47:49.746 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-16T09:47:49.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.g 2024-03-16T09:47:50.009 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:47:49 smithi077 systemd[1]: Stopping Ceph mds.g for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:50.508 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:47:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g[73913]: 2024-03-16T09:47:50.066+0000 7fc00f56c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:50.509 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:47:50 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g[73913]: 2024-03-16T09:47:50.066+0000 7fc00f56c700 -1 mds.g *** got signal Terminated *** 2024-03-16T09:47:53.508 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:47:53 smithi077 podman[92912]: 2024-03-16 09:47:53.232425075 +0000 UTC m=+3.275634527 container died b8e1b98ba418613b54111c4d8e83693c4053715e5d811ec4886dc21d59f41390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:47:53.878 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:47:53 smithi077 podman[92912]: 2024-03-16 09:47:53.674954846 +0000 UTC m=+3.718164308 container cleanup b8e1b98ba418613b54111c4d8e83693c4053715e5d811ec4886dc21d59f41390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:47:53.878 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 09:47:53 smithi077 bash[92912]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-g 2024-03-16T09:47:54.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.g.service' 2024-03-16T09:47:54.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:54.134 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-16T09:47:54.134 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-16T09:47:54.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.j 2024-03-16T09:47:54.480 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:47:54 smithi077 systemd[1]: Stopping Ceph mds.j for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:54.758 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:47:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j[72967]: 2024-03-16T09:47:54.478+0000 7f34569e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:54.759 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:47:54 smithi077 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j[72967]: 2024-03-16T09:47:54.478+0000 7f34569e9700 -1 mds.j *** got signal Terminated *** 2024-03-16T09:47:58.506 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:47:58 smithi077 podman[93033]: 2024-03-16 09:47:58.228581752 +0000 UTC m=+3.880764078 container died 20aa64164ac26d48e5d129b7a07092f5e1dea3cbb116429839d1d68567ed349f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:47:58.758 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:47:58 smithi077 podman[93033]: 2024-03-16 09:47:58.570366476 +0000 UTC m=+4.222548876 container cleanup 20aa64164ac26d48e5d129b7a07092f5e1dea3cbb116429839d1d68567ed349f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:47:58.759 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:47:58 smithi077 bash[93033]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j 2024-03-16T09:47:58.759 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 09:47:58 smithi077 podman[93047]: 2024-03-16 09:47:58.738021819 +0000 UTC m=+0.504497887 container remove 20aa64164ac26d48e5d129b7a07092f5e1dea3cbb116429839d1d68567ed349f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T09:47:58.948 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.j.service' 2024-03-16T09:47:58.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:47:58.982 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-16T09:47:58.983 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-16T09:47:58.983 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.b 2024-03-16T09:47:59.311 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:47:59 smithi119 systemd[1]: Stopping Ceph mds.b for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:47:59.689 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:47:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: 2024-03-16T09:47:59.308+0000 7f7af0251700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:47:59.690 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:47:59 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b[66043]: 2024-03-16T09:47:59.308+0000 7f7af0251700 -1 mds.b *** got signal Terminated *** 2024-03-16T09:48:01.344 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:48:01 smithi119 podman[70203]: 2024-03-16 09:48:01.041922797 +0000 UTC m=+1.857565289 container died ccf45c290e5ac6db618ab9c8a06e7ee690dfe21615382df7b324a37d7bd9eb81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T09:48:01.652 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:48:01 smithi119 podman[70203]: 2024-03-16 09:48:01.433509995 +0000 UTC m=+2.249152486 container cleanup ccf45c290e5ac6db618ab9c8a06e7ee690dfe21615382df7b324a37d7bd9eb81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-16T09:48:01.653 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 16 09:48:01 smithi119 bash[70203]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-b 2024-03-16T09:48:01.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.b.service' 2024-03-16T09:48:01.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:01.822 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-16T09:48:01.822 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-16T09:48:01.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.e 2024-03-16T09:48:02.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:48:01 smithi119 systemd[1]: Stopping Ceph mds.e for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:48:02.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:48:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e[65415]: 2024-03-16T09:48:02.128+0000 7f86506c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:48:02.190 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:48:02 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e[65415]: 2024-03-16T09:48:02.128+0000 7f86506c7700 -1 mds.e *** got signal Terminated *** 2024-03-16T09:48:05.306 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:48:05 smithi119 podman[70337]: 2024-03-16 09:48:05.039225435 +0000 UTC m=+2.995123917 container died 99588bee2562e8bd378e9f5179327c2b9aee61681c966083e143ae128a7bb1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T09:48:05.689 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:48:05 smithi119 podman[70337]: 2024-03-16 09:48:05.395535552 +0000 UTC m=+3.351434030 container cleanup 99588bee2562e8bd378e9f5179327c2b9aee61681c966083e143ae128a7bb1b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T09:48:05.690 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 16 09:48:05 smithi119 bash[70337]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-e 2024-03-16T09:48:05.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.e.service' 2024-03-16T09:48:05.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:05.894 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-16T09:48:05.894 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-16T09:48:05.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.h 2024-03-16T09:48:06.440 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:48:05 smithi119 systemd[1]: Stopping Ceph mds.h for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:48:06.440 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:48:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h[65720]: 2024-03-16T09:48:06.215+0000 7fd0247b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:48:06.440 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:48:06 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h[65720]: 2024-03-16T09:48:06.215+0000 7fd0247b8700 -1 mds.h *** got signal Terminated *** 2024-03-16T09:48:09.343 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:48:09 smithi119 podman[70459]: 2024-03-16 09:48:09.037409399 +0000 UTC m=+2.924293233 container died 61a7a126692ec6998ce84b93e9165f38642f493611e365948475fbaa6a0dbfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T09:48:09.651 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:48:09 smithi119 podman[70459]: 2024-03-16 09:48:09.416057431 +0000 UTC m=+3.302941272 container cleanup 61a7a126692ec6998ce84b93e9165f38642f493611e365948475fbaa6a0dbfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T09:48:09.652 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 16 09:48:09 smithi119 bash[70459]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-h 2024-03-16T09:48:09.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.h.service' 2024-03-16T09:48:09.834 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:09.834 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-16T09:48:09.834 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-16T09:48:09.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.k 2024-03-16T09:48:10.189 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:09 smithi119 systemd[1]: Stopping Ceph mds.k for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:48:10.189 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:10 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k[65092]: 2024-03-16T09:48:10.144+0000 7fb857a7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:48:10.189 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:10 smithi119 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k[65092]: 2024-03-16T09:48:10.144+0000 7fb857a7c700 -1 mds.k *** got signal Terminated *** 2024-03-16T09:48:11.864 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:11 smithi119 podman[70580]: 2024-03-16 09:48:11.576843271 +0000 UTC m=+1.529583483 container died 7002a90ce45b5e722ee9d2ebc0396cf5e81e7f0ef0449220c5dafb0c309749f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T09:48:12.189 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:11 smithi119 podman[70580]: 2024-03-16 09:48:11.944720466 +0000 UTC m=+1.897460678 container cleanup 7002a90ce45b5e722ee9d2ebc0396cf5e81e7f0ef0449220c5dafb0c309749f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:48:12.189 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:11 smithi119 bash[70580]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k 2024-03-16T09:48:12.324 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:12 smithi119 podman[70594]: 2024-03-16 09:48:12.204096684 +0000 UTC m=+0.627385804 container remove 7002a90ce45b5e722ee9d2ebc0396cf5e81e7f0ef0449220c5dafb0c309749f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-16T09:48:12.324 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:12 smithi119 systemd[1]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-16T09:48:12.324 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 16 09:48:12 smithi119 systemd[1]: Stopped Ceph mds.k for e287e1d6-e377-11ee-95c9-87774f69a715. 2024-03-16T09:48:12.325 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.k.service' 2024-03-16T09:48:12.359 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:12.359 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-16T09:48:12.359 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-16T09:48:12.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.c 2024-03-16T09:48:12.655 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:48:12 smithi129 systemd[1]: Stopping Ceph mds.c for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:48:12.918 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:48:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c[66464]: 2024-03-16T09:48:12.653+0000 7f0893189700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:48:12.918 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:48:12 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c[66464]: 2024-03-16T09:48:12.653+0000 7f0893189700 -1 mds.c *** got signal Terminated *** 2024-03-16T09:48:16.963 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:48:16 smithi129 podman[71021]: 2024-03-16 09:48:16.662071677 +0000 UTC m=+4.113699340 container died 6d306348933d820bdf46ae885d187614e22f901265107c449b2758382c847930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T09:48:17.297 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:48:17 smithi129 podman[71021]: 2024-03-16 09:48:17.061079793 +0000 UTC m=+4.512707459 container cleanup 6d306348933d820bdf46ae885d187614e22f901265107c449b2758382c847930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T09:48:17.297 INFO:journalctl@ceph.mds.c.smithi129.stdout:Mar 16 09:48:17 smithi129 bash[71021]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-c 2024-03-16T09:48:17.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.c.service' 2024-03-16T09:48:17.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:17.480 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-16T09:48:17.480 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-16T09:48:17.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.f 2024-03-16T09:48:17.851 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:48:17 smithi129 systemd[1]: Stopping Ceph mds.f for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:48:18.300 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:48:17 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f[66155]: 2024-03-16T09:48:17.848+0000 7fe46a995700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:48:18.300 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:48:17 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f[66155]: 2024-03-16T09:48:17.848+0000 7fe46a995700 -1 mds.f *** got signal Terminated *** 2024-03-16T09:48:21.300 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:48:21 smithi129 podman[71155]: 2024-03-16 09:48:21.038952905 +0000 UTC m=+3.333426410 container died 8c6964a9454733dd17a5297658b8e8d441e5dc8c862b5c38a6dacc75cc2a6c07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T09:48:21.750 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:48:21 smithi129 podman[71155]: 2024-03-16 09:48:21.498451 +0000 UTC m=+3.792924516 container cleanup 8c6964a9454733dd17a5297658b8e8d441e5dc8c862b5c38a6dacc75cc2a6c07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T09:48:21.751 INFO:journalctl@ceph.mds.f.smithi129.stdout:Mar 16 09:48:21 smithi129 bash[71155]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-f 2024-03-16T09:48:21.925 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.f.service' 2024-03-16T09:48:21.960 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:21.960 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-16T09:48:21.960 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-16T09:48:21.960 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.i 2024-03-16T09:48:22.300 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:48:22 smithi129 systemd[1]: Stopping Ceph mds.i for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:48:22.300 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:48:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i[66771]: 2024-03-16T09:48:22.258+0000 7fe06ca1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:48:22.300 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:48:22 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i[66771]: 2024-03-16T09:48:22.258+0000 7fe06ca1e700 -1 mds.i *** got signal Terminated *** 2024-03-16T09:48:25.300 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:48:25 smithi129 podman[71275]: 2024-03-16 09:48:25.034674927 +0000 UTC m=+2.863390710 container died 2569ce7aa2f2e4d6948b50ce59db3e312eb11bd7ca8b6dcba4fd791dd719f1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T09:48:25.587 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:48:25 smithi129 podman[71275]: 2024-03-16 09:48:25.385025269 +0000 UTC m=+3.213741057 container cleanup 2569ce7aa2f2e4d6948b50ce59db3e312eb11bd7ca8b6dcba4fd791dd719f1e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T09:48:25.587 INFO:journalctl@ceph.mds.i.smithi129.stdout:Mar 16 09:48:25 smithi129 bash[71275]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-i 2024-03-16T09:48:25.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.i.service' 2024-03-16T09:48:25.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:25.841 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-16T09:48:25.841 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-16T09:48:25.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.l 2024-03-16T09:48:26.165 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:48:25 smithi129 systemd[1]: Stopping Ceph mds.l for e287e1d6-e377-11ee-95c9-87774f69a715... 2024-03-16T09:48:26.550 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:48:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l[65854]: 2024-03-16T09:48:26.162+0000 7ffb264e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T09:48:26.550 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:48:26 smithi129 ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l[65854]: 2024-03-16T09:48:26.162+0000 7ffb264e0700 -1 mds.l *** got signal Terminated *** 2024-03-16T09:48:29.050 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:48:28 smithi129 podman[71398]: 2024-03-16 09:48:28.785941302 +0000 UTC m=+2.729643860 container died 292d32117075fdcde0a4ca2ccfd71b162a1fd691fe6dfafa5305f64b088b4fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-16T09:48:29.399 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:48:29 smithi129 podman[71398]: 2024-03-16 09:48:29.13063372 +0000 UTC m=+3.074336308 container cleanup 292d32117075fdcde0a4ca2ccfd71b162a1fd691fe6dfafa5305f64b088b4fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T09:48:29.400 INFO:journalctl@ceph.mds.l.smithi129.stdout:Mar 16 09:48:29 smithi129 bash[71398]: ceph-e287e1d6-e377-11ee-95c9-87774f69a715-mds-l 2024-03-16T09:48:29.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e287e1d6-e377-11ee-95c9-87774f69a715@mds.l.service' 2024-03-16T09:48:29.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T09:48:29.593 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-16T09:48:29.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e287e1d6-e377-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T09:48:29.785 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:48:32.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e287e1d6-e377-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T09:48:32.396 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:48:34.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e287e1d6-e377-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T09:48:35.015 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:48:37.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:48:37.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:48:37.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T09:48:37.420 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-16T09:48:37.423 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi077/crash 2024-03-16T09:48:37.424 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash -- . 2024-03-16T09:48:37.462 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T09:48:37.462 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-03-16T09:48:37.464 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi119/crash 2024-03-16T09:48:37.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash -- . 2024-03-16T09:48:37.611 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T09:48:37.611 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-03-16T09:48:37.613 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi129/crash 2024-03-16T09:48:37.614 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash -- . 2024-03-16T09:48:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T09:48:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-03-16T09:48:37.653 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-16T09:48:37.654 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T09:48:37.691 INFO:teuthology.orchestra.run.smithi077.stdout:2024-03-16T09:35:50.655146+0000 mon.a (mon.0) 487 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-16T09:48:37.692 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-16T09:48:37.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[SEC\]' /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T09:48:37.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]' /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T09:48:37.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[WRN\]' /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T09:48:37.916 INFO:teuthology.orchestra.run.smithi077.stdout:2024-03-16T09:35:50.655146+0000 mon.a (mon.0) 487 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-16T09:48:37.917 INFO:tasks.cephadm:Compressing logs... 2024-03-16T09:48:37.917 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T09:48:37.959 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T09:48:37.962 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T09:48:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T09:48:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:: No such file or directory 2024-03-16T09:48:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-16T09:48:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-16T09:48:37.985 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-16T09:48:37.987 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T09:48:37.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-16T09:48:37.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.a.log: /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-16T09:48:37.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-16T09:48:37.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-16T09:48:37.989 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.3.log: 90.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T09:48:37.990 INFO:teuthology.orchestra.run.smithi077.stderr: 94.7% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-16T09:48:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-16T09:48:37.990 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T09:48:37.990 INFO:teuthology.orchestra.run.smithi077.stderr: 94.8% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.j.log.gz/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.0.log: 2024-03-16T09:48:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log 2024-03-16T09:48:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log: 83.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T09:48:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T09:48:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T09:48:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-16T09:48:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log: 86.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T09:48:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-16T09:48:38.005 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:: No such file or directory 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:gzip/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.b.log: -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T09:48:38.009 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.1.log: /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.e.log: 94.8% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.y.log: 92.2% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log: 86.2% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T09:48:38.010 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:: No such file or directory 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.i.log: /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T09:48:38.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.5% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log: 94.8% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr: 82.6% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.2.log: 86.2% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr: 92.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T09:48:38.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-16T09:48:38.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.d.log: 94.7% 91.0% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T09:48:38.013 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-16T09:48:38.016 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-16T09:48:38.021 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T09:48:38.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.c.log: 94.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-16T09:48:38.033 INFO:teuthology.orchestra.run.smithi129.stderr: 92.5% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-16T09:48:38.034 INFO:teuthology.orchestra.run.smithi129.stderr: 90.9% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T09:48:38.040 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.4% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-16T09:48:38.048 INFO:teuthology.orchestra.run.smithi129.stderr: 94.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-16T09:48:38.058 INFO:teuthology.orchestra.run.smithi119.stderr: 93.0% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-16T09:48:38.185 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-16T09:48:38.201 INFO:teuthology.orchestra.run.smithi119.stderr: 92.5% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-16T09:48:38.268 INFO:teuthology.orchestra.run.smithi119.stderr: 91.9% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-16T09:48:38.284 INFO:teuthology.orchestra.run.smithi129.stderr: 91.8% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-16T09:48:38.399 INFO:teuthology.orchestra.run.smithi077.stderr: 92.9% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-16T09:48:38.488 INFO:teuthology.orchestra.run.smithi129.stderr: 93.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-16T09:48:38.546 INFO:teuthology.orchestra.run.smithi129.stderr: 93.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-16T09:48:38.559 INFO:teuthology.orchestra.run.smithi119.stderr: 93.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-16T09:48:38.602 INFO:teuthology.orchestra.run.smithi129.stderr: 93.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-16T09:48:38.615 INFO:teuthology.orchestra.run.smithi119.stderr: 93.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-16T09:48:38.643 INFO:teuthology.orchestra.run.smithi129.stderr: 93.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-16T09:48:38.645 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-16T09:48:38.645 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m0.669s 2024-03-16T09:48:38.645 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m2.618s 2024-03-16T09:48:38.646 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m0.140s 2024-03-16T09:48:38.665 INFO:teuthology.orchestra.run.smithi119.stderr: 93.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-16T09:48:38.677 INFO:teuthology.orchestra.run.smithi077.stderr: 93.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-16T09:48:38.731 INFO:teuthology.orchestra.run.smithi119.stderr: 93.3% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-16T09:48:38.733 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-16T09:48:38.733 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m0.757s 2024-03-16T09:48:38.733 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m3.106s 2024-03-16T09:48:38.733 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m0.125s 2024-03-16T09:48:38.733 INFO:teuthology.orchestra.run.smithi077.stderr: 93.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-16T09:48:38.760 INFO:teuthology.orchestra.run.smithi077.stderr: 90.6% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-16T09:48:38.782 INFO:teuthology.orchestra.run.smithi077.stderr: 93.1% -- replaced with /var/log/ceph/e287e1d6-e377-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-16T09:48:45.579 INFO:teuthology.orchestra.run.smithi077.stderr: 96.8% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-16T09:48:45.580 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-16T09:48:45.581 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m7.609s 2024-03-16T09:48:45.581 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m10.828s 2024-03-16T09:48:45.581 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.420s 2024-03-16T09:48:45.581 INFO:tasks.cephadm:Archiving logs... 2024-03-16T09:48:45.581 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi077/log 2024-03-16T09:48:45.582 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T09:48:46.854 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi119/log 2024-03-16T09:48:46.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T09:48:47.318 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi129/log 2024-03-16T09:48:47.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T09:48:47.722 INFO:tasks.cephadm:Removing cluster... 2024-03-16T09:48:47.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e287e1d6-e377-11ee-95c9-87774f69a715 --force 2024-03-16T09:48:47.904 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:48:48.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e287e1d6-e377-11ee-95c9-87774f69a715 --force 2024-03-16T09:48:48.349 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:48:48.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e287e1d6-e377-11ee-95c9-87774f69a715 --force 2024-03-16T09:48:48.789 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: e287e1d6-e377-11ee-95c9-87774f69a715 2024-03-16T09:48:49.046 INFO:tasks.cephadm:Removing cephadm ... 2024-03-16T09:48:49.046 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T09:48:49.064 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T09:48:49.080 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T09:48:49.095 INFO:tasks.cephadm:Teardown complete 2024-03-16T09:48:49.095 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-16T09:48:49.107 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-16T09:48:49.107 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T09:48:49.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T09:48:49.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T09:48:49.234 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T09:48:49.244 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T09:48:49.244 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T09:48:49.247 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T09:48:49.249 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T09:48:49.260 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-16T09:48:49.261 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-16T09:48:49.263 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-16T09:48:49.264 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T09:48:49.264 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-16T09:48:49.264 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +2404us[+2404us] +/- 103ms 2024-03-16T09:48:49.264 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -1613us[-1596us] +/- 79ms 2024-03-16T09:48:49.265 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -163us[ -146us] +/- 72ms 2024-03-16T09:48:49.265 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:48:49.266 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T09:48:49.266 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-16T09:48:49.266 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +2393us[+2393us] +/- 103ms 2024-03-16T09:48:49.266 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -1636us[-1636us] +/- 79ms 2024-03-16T09:48:49.267 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -191us[ -191us] +/- 72ms 2024-03-16T09:48:49.267 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:48:49.267 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T09:48:49.268 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-16T09:48:49.268 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +2406us[+2418us] +/- 103ms 2024-03-16T09:48:49.268 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -1627us[-1615us] +/- 79ms 2024-03-16T09:48:49.268 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -178us[ -166us] +/- 72ms 2024-03-16T09:48:49.268 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T09:48:49.268 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T09:48:49.279 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T09:48:49.279 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T09:48:49.290 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T09:48:49.387 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T09:48:49.459 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T09:48:49.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:48:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:48:49.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:48:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:48:49.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:48:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:48:49.620 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T09:48:49.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T09:48:49.640 INFO:teuthology.task.internal:Duration was 1432.795896 seconds 2024-03-16T09:48:49.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T09:48:49.651 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T09:48:49.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T09:48:49.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T09:48:49.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T09:48:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:48:49.696 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:48:49.703 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:48:50.053 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T09:48:50.053 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:48:50.054 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T09:48:50.114 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-03-16T09:48:50.114 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T09:48:50.142 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-03-16T09:48:50.143 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T09:48:50.170 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T09:48:50.170 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T09:48:50.173 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T09:48:50.185 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T09:48:50.239 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T09:48:50.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T09:48:50.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T09:48:50.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T09:48:50.664 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T09:48:50.675 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T09:48:50.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T09:48:50.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T09:48:50.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T09:48:50.758 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T09:48:50.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T09:48:50.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T09:48:50.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T09:48:50.795 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-03-16T09:48:50.798 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-03-16T09:48:50.824 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-03-16T09:48:50.844 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T09:48:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:48:50.865 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T09:48:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:48:50.879 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T09:48:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:48:50.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T09:48:50.912 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T09:48:50.912 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi077 2024-03-16T09:48:50.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T09:48:50.956 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi119 2024-03-16T09:48:50.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T09:48:50.993 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339/remote/smithi129 2024-03-16T09:48:50.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T09:48:51.031 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T09:48:51.031 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T09:48:51.033 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T09:48:51.037 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T09:48:51.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T09:48:51.092 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T09:48:51.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T09:48:51.102 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T09:48:51.102 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T09:48:51.104 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T09:48:51.107 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T09:48:51.116 INFO:teuthology.orchestra.run.smithi077.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 09:48 /home/ubuntu/cephtest 2024-03-16T09:48:51.118 INFO:teuthology.orchestra.run.smithi119.stdout: 266037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 09:48 /home/ubuntu/cephtest 2024-03-16T09:48:51.134 INFO:teuthology.orchestra.run.smithi129.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 09:48 /home/ubuntu/cephtest 2024-03-16T09:48:51.135 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T09:48:51.147 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T09:48:51.171 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T09:48:51.197 DEBUG:teuthology.nuke:shortname: smithi077 2024-03-16T09:48:51.197 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T09:48:51.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:15:11.667351', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLTS8r9NdLFvN0K4BbM6uhE+yI9Nh0KNrEGT9++u0nHPXqlYpk/PUO4lBpNqgbd0oEbrzFrhywZxP2HhLpnwdBM='} 2024-03-16T09:48:51.225 DEBUG:teuthology.nuke:shortname: smithi119 2024-03-16T09:48:51.226 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T09:48:51.251 DEBUG:teuthology.nuke:shortname: smithi129 2024-03-16T09:48:51.251 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T09:48:51.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:15:11.665921', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEBIe/wTNnCVhVXqDA4I9ZuwtHYwjcOZZNsln7CQLWslzBgc6wxMD0ro2uDDtUQ4MIdroV0ko3yVkEbTt1jSivM='} 2024-03-16T09:48:51.278 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:15:11.668586', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn+HU31qDJ+LsuVLv+K9fHW3uyynuHEGkCeum5Z5T5qfWVPTL7G/mSYcHZ6fMPiBCF6H+MCPYnEhKcnx8mNWTY='} 2024-03-16T09:48:51.288 INFO:teuthology.orchestra.console.smithi077:Power off 2024-03-16T09:48:51.288 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T09:48:51.311 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-03-16T09:48:51.311 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:51.326 INFO:teuthology.orchestra.console.smithi119:Power off 2024-03-16T09:48:51.326 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T09:48:51.340 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-16T09:48:51.349 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-03-16T09:48:51.349 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:51.368 INFO:teuthology.orchestra.console.smithi129:Power off 2024-03-16T09:48:51.368 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T09:48:51.378 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-16T09:48:51.390 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-03-16T09:48:51.391 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:51.413 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-16T09:48:55.341 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:55.379 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:55.415 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:55.452 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-16T09:48:55.488 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-16T09:48:55.524 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-16T09:48:59.454 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:59.489 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:59.524 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:48:59.565 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-03-16T09:48:59.598 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-16T09:48:59.633 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-03-16T09:48:59.666 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-03-16T09:48:59.734 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-03-16T09:49:03.603 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T09:49:03.712 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-03-16T09:49:03.812 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-03-16T09:49:03.917 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} duration: 1432.795895576477 failure_reason: '"2024-03-16T09:35:50.655146+0000 mon.a (mon.0) 487 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-16T09:49:03.918 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T09:49:04.020 INFO:teuthology.run:FAIL